Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240507 08:11:45.954913 5083 test_util.cc:268] Using random seed: -1129651144
2024-05-07T08:11:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-07T08:11:45Z Disabled control of system clock
I20240507 08:11:45.987229 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.254:37179
--webserver_interface=127.4.246.254
--webserver_port=0
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20240507 08:11:46.297179 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:46.297503 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:46.297673 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:46.304975 5091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 08:11:46.305081 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:46.305140 5091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 08:11:46.305228 5091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 08:11:46.314291 5091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.254:37179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.246.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:46.315672 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:46.317396 5091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:46.327965 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:46.328723 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:46.329401 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:46.330264 5091 server_base.cc:1034] running on GCE node
I20240507 08:11:46.331039 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:46.332378 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:46.333644 5091 hybrid_clock.cc:648] HybridClock initialized: now 1715069506333539 us; error 45 us; skew 500 ppm
I20240507 08:11:46.333986 5091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:46.336191 5091 webserver.cc:460] Webserver started at http://127.4.246.254:41413/ using document root <none> and password file <none>
I20240507 08:11:46.336902 5091 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:46.337033 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:46.337303 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:46.339877 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data/instance:
uuid: "a91047a8a5414eff9c06ca9ca2b548e8"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.340576 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal/instance:
uuid: "a91047a8a5414eff9c06ca9ca2b548e8"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.345450 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20240507 08:11:46.348652 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:46.350028 5091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240507 08:11:46.350194 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal
uuid: "a91047a8a5414eff9c06ca9ca2b548e8"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.350383 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:46.379504 5091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:46.380343 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:46.380628 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:46.398108 5091 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.254:37179
I20240507 08:11:46.398149 5156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.254:37179 every 8 connection(s)
I20240507 08:11:46.399418 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/data/info.pb
I20240507 08:11:46.402606 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5091
I20240507 08:11:46.402945 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-0/wal/instance
I20240507 08:11:46.403453 5157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:46.406236 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--webserver_interface=127.4.246.253
--webserver_port=0
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240507 08:11:46.411769 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:46.426820 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
W20240507 08:11:46.427901 5108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
W20240507 08:11:46.429425 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
I20240507 08:11:46.478752 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } attempt: 1
W20240507 08:11:46.480628 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
I20240507 08:11:46.548873 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } attempt: 2
W20240507 08:11:46.550928 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
I20240507 08:11:46.642172 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } attempt: 3
W20240507 08:11:46.644021 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
W20240507 08:11:46.720803 5159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:46.721124 5159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:46.721263 5159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:46.728638 5159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 08:11:46.728755 5159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:46.728837 5159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 08:11:46.728921 5159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 08:11:46.737884 5159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.246.253
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:46.739221 5159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:46.740939 5159 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:46.751156 5167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:46.751787 5170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:46.752645 5168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:46.753289 5159 server_base.cc:1034] running on GCE node
I20240507 08:11:46.754061 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:46.755462 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:46.756693 5159 hybrid_clock.cc:648] HybridClock initialized: now 1715069506756618 us; error 54 us; skew 500 ppm
I20240507 08:11:46.757028 5159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:46.764591 5159 webserver.cc:460] Webserver started at http://127.4.246.253:40743/ using document root <none> and password file <none>
I20240507 08:11:46.765329 5159 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:46.765460 5159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:46.765779 5159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:46.768427 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/instance:
uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.769228 5159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal/instance:
uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.774168 5159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240507 08:11:46.777369 5176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:46.778622 5159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:46.778782 5159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:46.778968 5159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:46.787240 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } attempt: 4
W20240507 08:11:46.788926 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
I20240507 08:11:46.824156 5159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:46.824975 5159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:46.825241 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:46.842612 5159 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.253:43459
I20240507 08:11:46.842631 5227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.253:43459 every 8 connection(s)
I20240507 08:11:46.844005 5159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
I20240507 08:11:46.847568 5228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:46.853077 5228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:46.854116 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5159
I20240507 08:11:46.854298 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal/instance
I20240507 08:11:46.855551 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.252:45297
--webserver_interface=127.4.246.252
--webserver_port=0
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240507 08:11:46.867000 5228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:46.876605 5228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } has no permanent_uuid. Determining permanent_uuid...
W20240507 08:11:46.877772 5180 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
W20240507 08:11:46.879158 5228 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:46.928426 5228 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 1
W20240507 08:11:46.930249 5228 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:46.998451 5228 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 2
W20240507 08:11:47.000154 5228 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:47.088173 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } attempt: 5
I20240507 08:11:47.091374 5228 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 3
W20240507 08:11:47.092922 5228 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:47.095137 5157 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } has no permanent_uuid. Determining permanent_uuid...
W20240507 08:11:47.096740 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:47.148030 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 1
W20240507 08:11:47.149839 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
W20240507 08:11:47.183888 5231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:47.184221 5231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:47.184360 5231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:47.191691 5231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 08:11:47.191829 5231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:47.191915 5231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 08:11:47.191999 5231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 08:11:47.201344 5231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.252:45297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.246.252
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:47.202729 5231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:47.204459 5231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:47.214815 5238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:47.215503 5241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:47.216236 5239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:47.217058 5231 server_base.cc:1034] running on GCE node
I20240507 08:11:47.217757 5231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:47.218036 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 2
I20240507 08:11:47.219308 5231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20240507 08:11:47.219784 5157 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:47.220539 5231 hybrid_clock.cc:648] HybridClock initialized: now 1715069507220471 us; error 48 us; skew 500 ppm
I20240507 08:11:47.220813 5231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:47.222908 5231 webserver.cc:460] Webserver started at http://127.4.246.252:34585/ using document root <none> and password file <none>
I20240507 08:11:47.223523 5231 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:47.223623 5231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:47.223906 5231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:47.226807 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data/instance:
uuid: "94c202eef7f3466fa3e703644b40879d"
format_stamp: "Formatted at 2024-05-07 08:11:47 on dist-test-slave-d8w8"
I20240507 08:11:47.227494 5231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal/instance:
uuid: "94c202eef7f3466fa3e703644b40879d"
format_stamp: "Formatted at 2024-05-07 08:11:47 on dist-test-slave-d8w8"
I20240507 08:11:47.232259 5231 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240507 08:11:47.235436 5247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:47.236142 5228 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 4
I20240507 08:11:47.236784 5231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:47.236968 5231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal
uuid: "94c202eef7f3466fa3e703644b40879d"
format_stamp: "Formatted at 2024-05-07 08:11:47 on dist-test-slave-d8w8"
I20240507 08:11:47.237135 5231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240507 08:11:47.237877 5228 consensus_peers.cc:644] Error getting permanent uuid from config peer 127.4.246.252:45297: Network error: Client connection negotiation failed: client connection to 127.4.246.252:45297: connect: Connection refused (error 111)
I20240507 08:11:47.256790 5231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:47.257620 5231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:47.257902 5231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:47.276304 5231 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.252:45297
I20240507 08:11:47.276326 5298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.252:45297 every 8 connection(s)
I20240507 08:11:47.277781 5231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/data/info.pb
I20240507 08:11:47.278147 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5231
I20240507 08:11:47.278358 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-2/wal/instance
I20240507 08:11:47.282558 5299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:47.289938 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:47.302358 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:47.309527 5299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } has no permanent_uuid. Determining permanent_uuid...
I20240507 08:11:47.319460 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Bootstrap starting.
I20240507 08:11:47.322831 5299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:47.323830 5299 log.cc:824] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:47.325999 5157 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 3
I20240507 08:11:47.326961 5299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: No bootstrap required, opened a new log
I20240507 08:11:47.335423 5299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.335815 5299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:47.335968 5299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c202eef7f3466fa3e703644b40879d, State: Initialized, Role: FOLLOWER
I20240507 08:11:47.336784 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: Bootstrap starting.
I20240507 08:11:47.336658 5299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.337563 5308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.337858 5308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.338868 5299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:11:47.340796 5157 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:47.342438 5157 log.cc:824] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:47.345129 5157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: No bootstrap required, opened a new log
W20240507 08:11:47.350171 5320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 08:11:47.350311 5320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 08:11:47.354467 5157 raft_consensus.cc:348] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.354838 5157 raft_consensus.cc:374] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:47.354972 5157 raft_consensus.cc:729] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a91047a8a5414eff9c06ca9ca2b548e8, State: Initialized, Role: FOLLOWER
I20240507 08:11:47.355641 5157 consensus_queue.cc:260] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.356530 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.356884 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.348244 5231 master_runner.cc:186] Error getting master registration for 127.4.246.254:37179: OK, instance_id { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" instance_seqno: 1715069506393219 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:11:47.357170 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:11:47.358228 5157 sys_catalog.cc:564] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: configured and running, proceeding with master startup.
W20240507 08:11:47.371798 5333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20240507 08:11:47.372071 5333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20240507 08:11:47.369509 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069506837755 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:11:47.378836 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240507 08:11:47.383740 5111 tablet.cc:2369] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:11:47.487619 5308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:47.487920 5308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.489785 5308 leader_election.cc:288] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a91047a8a5414eff9c06ca9ca2b548e8 (127.4.246.254:37179), a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459)
I20240507 08:11:47.490504 5132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94c202eef7f3466fa3e703644b40879d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" is_pre_election: true
I20240507 08:11:47.491086 5132 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c202eef7f3466fa3e703644b40879d in term 0.
I20240507 08:11:47.491654 5249 leader_election.cc:302] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c202eef7f3466fa3e703644b40879d, a91047a8a5414eff9c06ca9ca2b548e8; no voters:
I20240507 08:11:47.492067 5308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:47.492218 5308 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:47.492317 5308 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:47.495465 5308 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.496440 5132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94c202eef7f3466fa3e703644b40879d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a91047a8a5414eff9c06ca9ca2b548e8"
I20240507 08:11:47.496634 5132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:47.496738 5308 leader_election.cc:288] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [CANDIDATE]: Term 1 election: Requested vote from peers a91047a8a5414eff9c06ca9ca2b548e8 (127.4.246.254:37179), a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459)
I20240507 08:11:47.499819 5132 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 94c202eef7f3466fa3e703644b40879d in term 1.
I20240507 08:11:47.500280 5249 leader_election.cc:302] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 94c202eef7f3466fa3e703644b40879d, a91047a8a5414eff9c06ca9ca2b548e8; no voters:
I20240507 08:11:47.500582 5308 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:47.501195 5308 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [term 1 LEADER]: Becoming Leader. State: Replica: 94c202eef7f3466fa3e703644b40879d, State: Running, Role: LEADER
I20240507 08:11:47.501755 5308 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
W20240507 08:11:47.509948 5253 tablet.cc:2369] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:11:47.515798 5335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c202eef7f3466fa3e703644b40879d. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.516227 5335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
I20240507 08:11:47.522994 5337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240507 08:11:47.525188 5337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240507 08:11:47.535249 5132 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Refusing update from remote peer 94c202eef7f3466fa3e703644b40879d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:47.536623 5335 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:47.537124 5228 consensus_peers.cc:654] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } attempt: 5
I20240507 08:11:47.558779 5228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap starting.
I20240507 08:11:47.564666 5228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:47.566915 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c202eef7f3466fa3e703644b40879d. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.567180 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.570602 5335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.571422 5335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
I20240507 08:11:47.572585 5228 log.cc:824] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:47.575251 5337 catalog_manager.cc:1347] Generated new cluster ID: 5ffc45d58e0243d3a7f54e418cabbd00
I20240507 08:11:47.575354 5337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240507 08:11:47.582319 5322 sys_catalog.cc:455] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.582577 5322 sys_catalog.cc:458] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.598142 5228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: No bootstrap required, opened a new log
I20240507 08:11:47.604876 5337 catalog_manager.cc:1370] Generated new certificate authority record
I20240507 08:11:47.606603 5337 catalog_manager.cc:1504] Loading token signing keys...
I20240507 08:11:47.620170 5228 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.620999 5228 raft_consensus.cc:374] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:47.621325 5228 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39de2a69ed4492aab4d7a232b09c1ae, State: Initialized, Role: FOLLOWER
I20240507 08:11:47.622740 5228 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:47.628024 5337 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Generated new TSK 0
I20240507 08:11:47.629009 5337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240507 08:11:47.633694 5202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94c202eef7f3466fa3e703644b40879d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
I20240507 08:11:47.634191 5203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "94c202eef7f3466fa3e703644b40879d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" is_pre_election: true
I20240507 08:11:47.635087 5203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 94c202eef7f3466fa3e703644b40879d in term 0.
I20240507 08:11:47.636514 5201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:47.637660 5344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.638059 5344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.640705 5201 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Refusing update from remote peer 94c202eef7f3466fa3e703644b40879d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:47.641223 5228 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:11:47.646232 5308 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240507 08:11:47.654978 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.246.193:0
--local_ip_for_outbound_sockets=127.4.246.193
--webserver_interface=127.4.246.193
--webserver_port=0
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240507 08:11:47.684882 5344 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c202eef7f3466fa3e703644b40879d. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.684868 5350 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:47.685281 5344 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.685299 5350 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:47.758939 5361 catalog_manager.cc:1259] Loaded cluster ID: 5ffc45d58e0243d3a7f54e418cabbd00
I20240507 08:11:47.759060 5361 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: loading cluster ID for follower catalog manager: success
I20240507 08:11:47.772805 5361 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: acquiring CA information for follower catalog manager: success
I20240507 08:11:47.775146 5361 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240507 08:11:48.099817 5347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:48.100104 5347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:48.100248 5347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:48.107812 5347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:48.108028 5347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.246.193
I20240507 08:11:48.117791 5347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.246.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.246.193
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.246.193
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:48.119169 5347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:48.120924 5347 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:48.134687 5368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:48.135375 5367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:48.135954 5370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:48.136226 5347 server_base.cc:1034] running on GCE node
I20240507 08:11:48.136845 5347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:48.137905 5347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:48.139165 5347 hybrid_clock.cc:648] HybridClock initialized: now 1715069508139106 us; error 41 us; skew 500 ppm
I20240507 08:11:48.139443 5347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:48.141918 5347 webserver.cc:460] Webserver started at http://127.4.246.193:35791/ using document root <none> and password file <none>
I20240507 08:11:48.143190 5347 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:48.143321 5347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:48.143606 5347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:48.146646 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data/instance:
uuid: "dc665657d5b34fd7aec884e0f2668e11"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.147635 5347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/instance:
uuid: "dc665657d5b34fd7aec884e0f2668e11"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.149170 5350 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c202eef7f3466fa3e703644b40879d)
I20240507 08:11:48.149413 5350 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:48.151499 5132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" is_pre_election: true
I20240507 08:11:48.151623 5350 leader_election.cc:288] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91047a8a5414eff9c06ca9ca2b548e8 (127.4.246.254:37179), 94c202eef7f3466fa3e703644b40879d (127.4.246.252:45297)
I20240507 08:11:48.152482 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "94c202eef7f3466fa3e703644b40879d" is_pre_election: true
I20240507 08:11:48.153301 5180 leader_election.cc:302] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a39de2a69ed4492aab4d7a232b09c1ae; no voters: 94c202eef7f3466fa3e703644b40879d, a91047a8a5414eff9c06ca9ca2b548e8
I20240507 08:11:48.153739 5350 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:11:48.154129 5347 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20240507 08:11:48.157732 5376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.159713 5347 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240507 08:11:48.159986 5347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal
uuid: "dc665657d5b34fd7aec884e0f2668e11"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.160162 5347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:48.199513 5308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:48.199925 5308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
I20240507 08:11:48.207870 5347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:48.208678 5347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:48.208956 5347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:48.209936 5347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:11:48.211452 5347 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:11:48.211522 5347 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.211637 5347 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:11:48.211692 5347 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.245064 5347 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.193:45985
I20240507 08:11:48.245106 5491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.193:45985 every 8 connection(s)
I20240507 08:11:48.246942 5347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/data/info.pb
I20240507 08:11:48.251390 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5347
I20240507 08:11:48.251624 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/instance
I20240507 08:11:48.261081 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.246.194:0
--local_ip_for_outbound_sockets=127.4.246.194
--webserver_interface=127.4.246.194
--webserver_port=0
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240507 08:11:48.284904 5493 heartbeater.cc:347] Connected to a master server at 127.4.246.254:37179
I20240507 08:11:48.285223 5493 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.286239 5493 heartbeater.cc:511] Master 127.4.246.254:37179 requested a full tablet report, sending...
I20240507 08:11:48.288551 5122 ts_manager.cc:194] Registered new tserver with Master: dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:48.291957 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:48.292102 5494 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.293004 5494 heartbeater.cc:511] Master 127.4.246.253:43459 requested a full tablet report, sending...
I20240507 08:11:48.293337 5492 heartbeater.cc:347] Connected to a master server at 127.4.246.252:45297
I20240507 08:11:48.293450 5492 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.293843 5492 heartbeater.cc:511] Master 127.4.246.252:45297 requested a full tablet report, sending...
I20240507 08:11:48.295341 5193 ts_manager.cc:194] Registered new tserver with Master: dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:48.295361 5264 ts_manager.cc:194] Registered new tserver with Master: dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:48.296872 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.246.193:55019
I20240507 08:11:48.374378 5333 catalog_manager.cc:1259] Loaded cluster ID: 5ffc45d58e0243d3a7f54e418cabbd00
I20240507 08:11:48.374498 5333 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: loading cluster ID for follower catalog manager: success
I20240507 08:11:48.377971 5333 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: acquiring CA information for follower catalog manager: success
I20240507 08:11:48.380227 5333 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20240507 08:11:48.613600 5502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:48.613884 5502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:48.614005 5502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:48.621268 5502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:48.621457 5502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.246.194
I20240507 08:11:48.631039 5502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.246.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.246.194
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.246.194
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:48.632305 5502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:48.634048 5502 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:48.646826 5508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:48.647570 5507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:48.647879 5510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:48.648536 5502 server_base.cc:1034] running on GCE node
I20240507 08:11:48.649148 5502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:48.650120 5502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:48.651340 5502 hybrid_clock.cc:648] HybridClock initialized: now 1715069508651273 us; error 41 us; skew 500 ppm
I20240507 08:11:48.651683 5502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:48.654039 5502 webserver.cc:460] Webserver started at http://127.4.246.194:44555/ using document root <none> and password file <none>
I20240507 08:11:48.654803 5502 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:48.654922 5502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:48.655181 5502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:48.658335 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data/instance:
uuid: "fe673739d8084e7da5d7edd722e857d8"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.659104 5502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/instance:
uuid: "fe673739d8084e7da5d7edd722e857d8"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.664685 5502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240507 08:11:48.668015 5516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.669313 5502 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240507 08:11:48.669478 5502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal
uuid: "fe673739d8084e7da5d7edd722e857d8"
format_stamp: "Formatted at 2024-05-07 08:11:48 on dist-test-slave-d8w8"
I20240507 08:11:48.669684 5502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:48.701673 5521 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c202eef7f3466fa3e703644b40879d)
I20240507 08:11:48.701855 5521 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:48.702724 5521 leader_election.cc:288] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91047a8a5414eff9c06ca9ca2b548e8 (127.4.246.254:37179), 94c202eef7f3466fa3e703644b40879d (127.4.246.252:45297)
I20240507 08:11:48.703177 5132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" is_pre_election: true
I20240507 08:11:48.703258 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "94c202eef7f3466fa3e703644b40879d" is_pre_election: true
I20240507 08:11:48.703855 5178 leader_election.cc:302] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a39de2a69ed4492aab4d7a232b09c1ae; no voters: 94c202eef7f3466fa3e703644b40879d, a91047a8a5414eff9c06ca9ca2b548e8
I20240507 08:11:48.704144 5521 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:11:48.710871 5502 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:48.711683 5502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:48.711982 5502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:48.712910 5502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:11:48.714468 5502 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:11:48.714538 5502 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.714659 5502 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:11:48.714713 5502 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:48.749097 5502 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.194:41443
I20240507 08:11:48.749173 5629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.194:41443 every 8 connection(s)
I20240507 08:11:48.750555 5502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/data/info.pb
I20240507 08:11:48.752779 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5502
I20240507 08:11:48.753018 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/instance
I20240507 08:11:48.759130 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.246.195:0
--local_ip_for_outbound_sockets=127.4.246.195
--webserver_interface=127.4.246.195
--webserver_port=0
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20240507 08:11:48.770794 5630 heartbeater.cc:347] Connected to a master server at 127.4.246.252:45297
I20240507 08:11:48.771106 5631 heartbeater.cc:347] Connected to a master server at 127.4.246.254:37179
I20240507 08:11:48.771198 5630 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.771250 5631 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.772508 5630 heartbeater.cc:511] Master 127.4.246.252:45297 requested a full tablet report, sending...
I20240507 08:11:48.772549 5631 heartbeater.cc:511] Master 127.4.246.254:37179 requested a full tablet report, sending...
I20240507 08:11:48.774370 5264 ts_manager.cc:194] Registered new tserver with Master: fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:48.774351 5122 ts_manager.cc:194] Registered new tserver with Master: fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:48.775269 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.246.194:35513
I20240507 08:11:48.775700 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:48.775830 5632 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:48.776213 5632 heartbeater.cc:511] Master 127.4.246.253:43459 requested a full tablet report, sending...
I20240507 08:11:48.777264 5193 ts_manager.cc:194] Registered new tserver with Master: fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:48.799170 5492 heartbeater.cc:503] Master 127.4.246.252:45297 was elected leader, sending a full tablet report...
I20240507 08:11:48.996073 5638 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c202eef7f3466fa3e703644b40879d)
I20240507 08:11:48.996282 5638 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:48.997476 5638 leader_election.cc:288] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459), 94c202eef7f3466fa3e703644b40879d (127.4.246.252:45297)
I20240507 08:11:48.998111 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "94c202eef7f3466fa3e703644b40879d" is_pre_election: true
I20240507 08:11:48.998112 5201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" is_pre_election: true
I20240507 08:11:48.998829 5108 leader_election.cc:302] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91047a8a5414eff9c06ca9ca2b548e8; no voters: 94c202eef7f3466fa3e703644b40879d, a39de2a69ed4492aab4d7a232b09c1ae
I20240507 08:11:48.999199 5638 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240507 08:11:49.098407 5637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:49.098678 5637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:49.098790 5637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240507 08:11:49.105878 5637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:49.106060 5637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.246.195
I20240507 08:11:49.115041 5637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.246.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.246.195
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.246.195
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:49.116286 5637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:49.117980 5637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240507 08:11:49.132712 5644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:49.133040 5643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:49.133728 5646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:49.134409 5637 server_base.cc:1034] running on GCE node
I20240507 08:11:49.135059 5637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:49.136202 5637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:49.137468 5637 hybrid_clock.cc:648] HybridClock initialized: now 1715069509137392 us; error 58 us; skew 500 ppm
I20240507 08:11:49.137786 5637 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:49.140149 5637 webserver.cc:460] Webserver started at http://127.4.246.195:42641/ using document root <none> and password file <none>
I20240507 08:11:49.141372 5637 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:49.141496 5637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:49.141793 5637 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240507 08:11:49.144428 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data/instance:
uuid: "1cbb887175f345ada2433adaec906fd4"
format_stamp: "Formatted at 2024-05-07 08:11:49 on dist-test-slave-d8w8"
I20240507 08:11:49.145076 5637 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/instance:
uuid: "1cbb887175f345ada2433adaec906fd4"
format_stamp: "Formatted at 2024-05-07 08:11:49 on dist-test-slave-d8w8"
I20240507 08:11:49.149935 5637 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20240507 08:11:49.153259 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:49.154618 5637 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:49.154785 5637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal
uuid: "1cbb887175f345ada2433adaec906fd4"
format_stamp: "Formatted at 2024-05-07 08:11:49 on dist-test-slave-d8w8"
I20240507 08:11:49.155007 5637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:49.176470 5637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:11:49.177238 5637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:11:49.177523 5637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:49.178545 5637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240507 08:11:49.179932 5637 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240507 08:11:49.180004 5637 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:49.180086 5637 ts_tablet_manager.cc:590] Registered 0 tablets
I20240507 08:11:49.180136 5637 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:49.213502 5637 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.195:44165
I20240507 08:11:49.213523 5764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.195:44165 every 8 connection(s)
I20240507 08:11:49.214824 5637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/data/info.pb
I20240507 08:11:49.220624 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5637
I20240507 08:11:49.220798 5083 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/instance
I20240507 08:11:49.232159 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:49.233042 5767 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:49.234153 5767 heartbeater.cc:511] Master 127.4.246.253:43459 requested a full tablet report, sending...
I20240507 08:11:49.235918 5193 ts_manager.cc:194] Registered new tserver with Master: 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:49.238050 5766 heartbeater.cc:347] Connected to a master server at 127.4.246.254:37179
I20240507 08:11:49.238076 5765 heartbeater.cc:347] Connected to a master server at 127.4.246.252:45297
I20240507 08:11:49.238200 5766 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:49.238216 5765 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:49.238701 5766 heartbeater.cc:511] Master 127.4.246.254:37179 requested a full tablet report, sending...
I20240507 08:11:49.238746 5765 heartbeater.cc:511] Master 127.4.246.252:45297 requested a full tablet report, sending...
I20240507 08:11:49.239657 5264 ts_manager.cc:194] Registered new tserver with Master: 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:49.239858 5122 ts_manager.cc:194] Registered new tserver with Master: 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:49.240676 5264 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.246.195:47089
I20240507 08:11:49.245363 5083 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20240507 08:11:49.276450 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_57c296abe8984fe4bf604ed45afc13b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.276474 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_09850ce797f641bf9ad48be7e511ff7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.278268 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09850ce797f641bf9ad48be7e511ff7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.279492 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57c296abe8984fe4bf604ed45afc13b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.280689 5630 heartbeater.cc:503] Master 127.4.246.252:45297 was elected leader, sending a full tablet report...
I20240507 08:11:49.282085 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9ac23f1d58d443b680d22e7adcf7898c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.282899 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ac23f1d58d443b680d22e7adcf7898c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.284878 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4f966ce31c3844fc870b246b911db02e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.289604 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5ad0a1a366f444beaf0d631a6a081efa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.290622 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eeca148e306a4c5083a0237bee7c9664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.291729 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_807737a384f94889baa10571f5493775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.292738 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6bfd81c4214f41da8b9afe2ada55ec31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.293897 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ef815067dce9448faa374118750a1933"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.285696 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f966ce31c3844fc870b246b911db02e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.296053 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d8840d69187f4dfeb07dffb8b035ac28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.296788 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8840d69187f4dfeb07dffb8b035ac28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.290278 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ad0a1a366f444beaf0d631a6a081efa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.291309 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eeca148e306a4c5083a0237bee7c9664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.292340 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_807737a384f94889baa10571f5493775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.293447 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bfd81c4214f41da8b9afe2ada55ec31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.294585 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef815067dce9448faa374118750a1933 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.347334 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_09850ce797f641bf9ad48be7e511ff7d" } new_table_name: "default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659" modify_external_catalogs: true
I20240507 08:11:49.350090 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_57c296abe8984fe4bf604ed45afc13b2" } new_table_name: "default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc" modify_external_catalogs: true
I20240507 08:11:49.365481 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5ad0a1a366f444beaf0d631a6a081efa" } modify_external_catalogs: true
I20240507 08:11:49.365832 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5ad0a1a366f444beaf0d631a6a081efa" } modify_external_catalogs: true
I20240507 08:11:49.365999 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4f966ce31c3844fc870b246b911db02e" } new_table_name: "default.Table_8da8971bb4e54275af1bd500580ef3c4" modify_external_catalogs: true
I20240507 08:11:49.367638 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9ac23f1d58d443b680d22e7adcf7898c" } new_table_name: "default.Table_b5e92e3768db4d958956b9730db5e8a8" modify_external_catalogs: true
I20240507 08:11:49.382880 5083 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrbFV97/build/asan/bin/kudu pid 5159
I20240507 08:11:49.384008 5223 generic_service.cc:196] Checking for leaks (request via RPC)
I20240507 08:11:49.390367 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_807737a384f94889baa10571f5493775" } modify_external_catalogs: true
I20240507 08:11:49.390617 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_807737a384f94889baa10571f5493775" } modify_external_catalogs: true
I20240507 08:11:49.391891 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6bfd81c4214f41da8b9afe2ada55ec31" } modify_external_catalogs: true
I20240507 08:11:49.391912 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ef815067dce9448faa374118750a1933" } new_table_name: "default.Table_5544196c0f57417e9ab15af0460ffcd8" modify_external_catalogs: true
I20240507 08:11:49.392083 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6bfd81c4214f41da8b9afe2ada55ec31" } modify_external_catalogs: true
I20240507 08:11:49.392493 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eeca148e306a4c5083a0237bee7c9664" } modify_external_catalogs: true
I20240507 08:11:49.393231 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eeca148e306a4c5083a0237bee7c9664" } modify_external_catalogs: true
W20240507 08:11:49.395814 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6d9ab293113410c857b5de903e21eb5 has no consensus state
I20240507 08:11:49.396449 5262 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 2207a4996d8d47b2b83d4619729f8ae3
W20240507 08:11:49.396651 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9922d86933cc40dda1efff9ef03dcdf2 has no consensus state
W20240507 08:11:49.396986 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0cd6fa62dcd848118d45e77d25dfd6d4 has no consensus state
I20240507 08:11:49.399225 5256 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 1aee440a01144a4984e2666901113e90
W20240507 08:11:49.400713 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86b8cfe64e5f418bbfb65087003beb66 has no consensus state
I20240507 08:11:49.401821 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc" } modify_external_catalogs: true
I20240507 08:11:49.402031 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc" } modify_external_catalogs: true
W20240507 08:11:49.431430 5249 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet d6d9ab293113410c857b5de903e21eb5 (table default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d6d9ab293113410c857b5de903e21eb5 consensus state has no leader
I20240507 08:11:49.438256 5257 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 0512dd116aad4687ad964fe1b615028f
W20240507 08:11:49.439198 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa1942369ba1413cb1c1198d4c972234 has no consensus state
I20240507 08:11:49.438747 5699 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9ab293113410c857b5de903e21eb5 (DEFAULT_TABLE table=default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.439761 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:49.440485 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9ab293113410c857b5de903e21eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.441879 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.442523 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd6fa62dcd848118d45e77d25dfd6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.441551 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.445756 5259 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 867d12a648074a5994aed184d16c3aa4
I20240507 08:11:49.461472 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.460143 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.461992 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a66fa7446a4dbea3885f8778e7f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.461823 5564 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9ab293113410c857b5de903e21eb5 (DEFAULT_TABLE table=default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.463304 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9ab293113410c857b5de903e21eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.463877 5562 tablet_service.cc:1467] Processing CreateTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.464433 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.464813 5561 tablet_service.cc:1467] Processing CreateTablet for tablet aa1942369ba1413cb1c1198d4c972234 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.465209 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.466348 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 (DEFAULT_TABLE table=default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.466491 5696 tablet_service.cc:1467] Processing CreateTablet for tablet aa1942369ba1413cb1c1198d4c972234 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.467180 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9922d86933cc40dda1efff9ef03dcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.468760 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a66fa7446a4dbea3885f8778e7f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.469614 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1942369ba1413cb1c1198d4c972234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.470326 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b8cfe64e5f418bbfb65087003beb66. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.471886 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d6d9ab293113410c857b5de903e21eb5 because the tablet was not found. No further retry: Not found: Tablet not found: d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:49.473776 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd6fa62dcd848118d45e77d25dfd6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.495280 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.498291 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 (DEFAULT_TABLE table=default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.501029 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b8cfe64e5f418bbfb65087003beb66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.500669 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.503079 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9922d86933cc40dda1efff9ef03dcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.502579 5423 tablet_service.cc:1467] Processing CreateTablet for tablet aa1942369ba1413cb1c1198d4c972234 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.504264 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1942369ba1413cb1c1198d4c972234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.505748 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.506429 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86b8cfe64e5f418bbfb65087003beb66. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.507035 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d6d9ab293113410c857b5de903e21eb5 because the tablet was not found. No further retry: Not found: Tablet not found: d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:49.507404 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa1942369ba1413cb1c1198d4c972234. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.510422 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.511058 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0cd6fa62dcd848118d45e77d25dfd6d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.513841 5839 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:49.518178 5839 tablet_bootstrap.cc:654] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.526849 5839 log.cc:824] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:49.502863 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 (DEFAULT_TABLE table=default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.528682 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9922d86933cc40dda1efff9ef03dcdf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.530339 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 86b8cfe64e5f418bbfb65087003beb66 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.530684 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet e4a66fa7446a4dbea3885f8778e7f92b from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.530766 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet aa1942369ba1413cb1c1198d4c972234 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.531391 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet e4a66fa7446a4dbea3885f8778e7f92b from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.531788 5426 tablet_service.cc:1467] Processing CreateTablet for tablet d6d9ab293113410c857b5de903e21eb5 (DEFAULT_TABLE table=default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.532037 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 0cd6fa62dcd848118d45e77d25dfd6d4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.532322 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6d9ab293113410c857b5de903e21eb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.537312 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 86b8cfe64e5f418bbfb65087003beb66
I20240507 08:11:49.537709 5424 tablet_service.cc:1467] Processing CreateTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.538295 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4a66fa7446a4dbea3885f8778e7f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.540555 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:49.541052 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 0cd6fa62dcd848118d45e77d25dfd6d4 deleted and replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:49.542380 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.542740 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:49.543232 5258 catalog_manager.cc:6554] ReplaceTablet: tablet e4a66fa7446a4dbea3885f8778e7f92b deleted and replaced by tablet 5cfe3c25a9a84eab9091502955453951
I20240507 08:11:49.543978 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.544229 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.544313 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:49.544325 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd6fa62dcd848118d45e77d25dfd6d4 already in progress: creating tablet
I20240507 08:11:49.544580 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.545130 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cdff15a4d49840fca28702f23f507243"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.545689 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdff15a4d49840fca28702f23f507243 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.545552 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_19215ebf25be44c88d9a815d887298d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.546237 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19215ebf25be44c88d9a815d887298d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.547096 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e4a66fa7446a4dbea3885f8778e7f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a66fa7446a4dbea3885f8778e7f92b already in progress: creating tablet
W20240507 08:11:49.547379 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd6fa62dcd848118d45e77d25dfd6d4 already in progress: creating tablet
I20240507 08:11:49.547938 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.547937 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 86b8cfe64e5f418bbfb65087003beb66 deleted and replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4
I20240507 08:11:49.548810 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aa1942369ba1413cb1c1198d4c972234
I20240507 08:11:49.549223 5257 catalog_manager.cc:6554] ReplaceTablet: tablet aa1942369ba1413cb1c1198d4c972234 deleted and replaced by tablet cbf73c218b944679a12d769b1929e535
I20240507 08:11:49.549441 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.551890 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.552115 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:49.552785 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 86b8cfe64e5f418bbfb65087003beb66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b8cfe64e5f418bbfb65087003beb66 already in progress: creating tablet
W20240507 08:11:49.553089 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aa1942369ba1413cb1c1198d4c972234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1942369ba1413cb1c1198d4c972234 already in progress: creating tablet
W20240507 08:11:49.554580 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e4a66fa7446a4dbea3885f8778e7f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a66fa7446a4dbea3885f8778e7f92b already in progress: creating tablet
I20240507 08:11:49.555090 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:49.555748 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 86b8cfe64e5f418bbfb65087003beb66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b8cfe64e5f418bbfb65087003beb66 already in progress: creating tablet
I20240507 08:11:49.556089 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:49.556689 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet aa1942369ba1413cb1c1198d4c972234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1942369ba1413cb1c1198d4c972234 already in progress: creating tablet
I20240507 08:11:49.558924 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.559445 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.560793 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_26f13967b3f2463688957c40bc77f155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.561290 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26f13967b3f2463688957c40bc77f155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.562681 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_345d6952a3384ee29f957390161d28dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.563273 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345d6952a3384ee29f957390161d28dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.566495 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.567127 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ffe98d424a4eb5a08be067f96a50f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.567370 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.567742 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ffe98d424a4eb5a08be067f96a50f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.566550 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.570533 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1b4843bfc4cb491092f9b3204d8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.570575 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 (DEFAULT_TABLE table=default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.570979 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34ffe98d424a4eb5a08be067f96a50f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.571683 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.572033 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1b4843bfc4cb491092f9b3204d8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.574316 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.574674 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ac1b4843bfc4cb491092f9b3204d8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.578773 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.580813 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_78072fd0f4b0453db900eb72f79ef48f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.581385 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78072fd0f4b0453db900eb72f79ef48f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.582640 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_217c38613c88457dbd818c27cee61f17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.583153 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_217c38613c88457dbd818c27cee61f17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.590109 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 717460af19164c058f42c7a03095bb30 (DEFAULT_TABLE table=default.Table_cdff15a4d49840fca28702f23f507243 [id=5d190d55ad4748cca262a1a33e12b68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.590520 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717460af19164c058f42c7a03095bb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.590795 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe3c25a9a84eab9091502955453951 (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.591012 5419 tablet_service.cc:1467] Processing CreateTablet for tablet cbf73c218b944679a12d769b1929e535 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.591151 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe3c25a9a84eab9091502955453951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.591320 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf73c218b944679a12d769b1929e535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.592367 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1a8c9d8abde548c1a409e34093aba24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.592967 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a8c9d8abde548c1a409e34093aba24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.594203 5694 tablet_service.cc:1467] Processing CreateTablet for tablet cbf73c218b944679a12d769b1929e535 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.594569 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf73c218b944679a12d769b1929e535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.596343 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe3c25a9a84eab9091502955453951 (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.596720 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe3c25a9a84eab9091502955453951. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.597952 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d8840d69187f4dfeb07dffb8b035ac28" } new_table_name: "default.Table_a1e20af4278a4c35ab581d660cc9ec3e" modify_external_catalogs: true
I20240507 08:11:49.598629 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5e92e3768db4d958956b9730db5e8a8" } modify_external_catalogs: true
I20240507 08:11:49.598819 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5e92e3768db4d958956b9730db5e8a8" } modify_external_catalogs: true
I20240507 08:11:49.599581 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_26f13967b3f2463688957c40bc77f155" } new_table_name: "default.Table_a873e078e67e44b4871044bbe3846aba" modify_external_catalogs: true
I20240507 08:11:49.601516 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 717460af19164c058f42c7a03095bb30 (DEFAULT_TABLE table=default.Table_cdff15a4d49840fca28702f23f507243 [id=5d190d55ad4748cca262a1a33e12b68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.601961 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 717460af19164c058f42c7a03095bb30 (DEFAULT_TABLE table=default.Table_cdff15a4d49840fca28702f23f507243 [id=5d190d55ad4748cca262a1a33e12b68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.601987 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cdff15a4d49840fca28702f23f507243" } new_table_name: "default.Table_8051479eeaf74dd8beccf13762a64249" modify_external_catalogs: true
I20240507 08:11:49.602310 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717460af19164c058f42c7a03095bb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.603404 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66" } modify_external_catalogs: true
I20240507 08:11:49.603602 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66" } modify_external_catalogs: true
I20240507 08:11:49.604672 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 717460af19164c058f42c7a03095bb30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.605101 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2e8bf6d3e14e49419acb16babc07dd38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.605695 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e8bf6d3e14e49419acb16babc07dd38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.608670 5843 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.611459 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b7024e203f50440095d2cf53ca83c775"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.611945 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7024e203f50440095d2cf53ca83c775 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:49.612610 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5cfe3c25a9a84eab9091502955453951 consensus state has no leader
W20240507 08:11:49.613046 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a485c1dc1cae40fcadce8ce75a5607b2 has no consensus state
I20240507 08:11:49.613152 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 5cfe3c25a9a84eab9091502955453951 (DEFAULT_TABLE table=default.Table_d8840d69187f4dfeb07dffb8b035ac28 [id=70026f101a8d4a1db79650e8e1f66c57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.613567 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cfe3c25a9a84eab9091502955453951. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.617238 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 717460af19164c058f42c7a03095bb30 consensus state has no leader
I20240507 08:11:49.617367 5843 tablet_bootstrap.cc:654] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.618813 5843 log.cc:824] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:49.622399 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 71bbe28a013344bcb6529247d7139ab9 (DEFAULT_TABLE table=default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.622645 5421 tablet_service.cc:1467] Processing CreateTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 (DEFAULT_TABLE table=default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.622764 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bbe28a013344bcb6529247d7139ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.622982 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a485c1dc1cae40fcadce8ce75a5607b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.623492 5420 tablet_service.cc:1467] Processing CreateTablet for tablet bc619db3a2944ecc9f0112b28de148a6 (DEFAULT_TABLE table=default.Table_19215ebf25be44c88d9a815d887298d2 [id=108648d0bbfe41cd961bdd8b7736989d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.623919 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc619db3a2944ecc9f0112b28de148a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.625873 5561 tablet_service.cc:1467] Processing CreateTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 (DEFAULT_TABLE table=default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.626060 5565 tablet_service.cc:1467] Processing CreateTablet for tablet cbf73c218b944679a12d769b1929e535 (DEFAULT_TABLE table=default.Table_5544196c0f57417e9ab15af0460ffcd8 [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.627300 5698 tablet_service.cc:1467] Processing CreateTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 (DEFAULT_TABLE table=default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.627694 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a485c1dc1cae40fcadce8ce75a5607b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.629494 5560 tablet_service.cc:1467] Processing CreateTablet for tablet bc619db3a2944ecc9f0112b28de148a6 (DEFAULT_TABLE table=default.Table_19215ebf25be44c88d9a815d887298d2 [id=108648d0bbfe41cd961bdd8b7736989d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.630851 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 (DEFAULT_TABLE table=default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.631273 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb62017acbd4472ac5b143eafb36ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.631846 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9922d86933cc40dda1efff9ef03dcdf2
I20240507 08:11:49.632122 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:49.632879 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.632978 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a485c1dc1cae40fcadce8ce75a5607b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.633157 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.633183 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.633486 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.633502 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc619db3a2944ecc9f0112b28de148a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.634835 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9922d86933cc40dda1efff9ef03dcdf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9922d86933cc40dda1efff9ef03dcdf2 already in progress: creating tablet
W20240507 08:11:49.635082 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 71bbe28a013344bcb6529247d7139ab9 because the tablet was not found. No further retry: Not found: Tablet not found: 71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:49.639226 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:49.640010 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 71bbe28a013344bcb6529247d7139ab9 because the tablet was not found. No further retry: Not found: Tablet not found: 71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:49.640103 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_78072fd0f4b0453db900eb72f79ef48f" } new_table_name: "default.Table_bd16315ec95647f699c27a2df5403d6e" modify_external_catalogs: true
I20240507 08:11:49.640362 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:49.640996 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9922d86933cc40dda1efff9ef03dcdf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9922d86933cc40dda1efff9ef03dcdf2 already in progress: creating tablet
I20240507 08:11:49.641528 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 71bbe28a013344bcb6529247d7139ab9 (DEFAULT_TABLE table=default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.641944 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bbe28a013344bcb6529247d7139ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.645823 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8051479eeaf74dd8beccf13762a64249" } modify_external_catalogs: true
I20240507 08:11:49.646003 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8051479eeaf74dd8beccf13762a64249" } modify_external_catalogs: true
I20240507 08:11:49.646070 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5544196c0f57417e9ab15af0460ffcd8" } new_table_name: "default.Table_a5900476f17d41e883656a38ce31f881" modify_external_catalogs: true
I20240507 08:11:49.647521 5840 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:49.647614 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1a8c9d8abde548c1a409e34093aba24d" } modify_external_catalogs: true
I20240507 08:11:49.647825 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1a8c9d8abde548c1a409e34093aba24d" } modify_external_catalogs: true
I20240507 08:11:49.649664 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 71bbe28a013344bcb6529247d7139ab9 (DEFAULT_TABLE table=default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.650131 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_217c38613c88457dbd818c27cee61f17" } modify_external_catalogs: true
I20240507 08:11:49.650336 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_217c38613c88457dbd818c27cee61f17" } modify_external_catalogs: true
I20240507 08:11:49.651787 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bbe28a013344bcb6529247d7139ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.652233 5840 tablet_bootstrap.cc:654] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:49.654031 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e8a046cce1f6467390071746a5e39fd9 has no consensus state
I20240507 08:11:49.655198 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 (DEFAULT_TABLE table=default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:49.656121 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d6d9ab293113410c857b5de903e21eb5 because the tablet was not found. No further retry: Not found: Tablet not found: d6d9ab293113410c857b5de903e21eb5
W20240507 08:11:49.656385 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e4a66fa7446a4dbea3885f8778e7f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4a66fa7446a4dbea3885f8778e7f92b already in progress: creating tablet
W20240507 08:11:49.656646 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 86b8cfe64e5f418bbfb65087003beb66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b8cfe64e5f418bbfb65087003beb66 already in progress: creating tablet
W20240507 08:11:49.656874 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet aa1942369ba1413cb1c1198d4c972234 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa1942369ba1413cb1c1198d4c972234 already in progress: creating tablet
W20240507 08:11:49.657104 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd6fa62dcd848118d45e77d25dfd6d4 already in progress: creating tablet
W20240507 08:11:49.657388 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9922d86933cc40dda1efff9ef03dcdf2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9922d86933cc40dda1efff9ef03dcdf2 already in progress: creating tablet
W20240507 08:11:49.657961 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 71bbe28a013344bcb6529247d7139ab9 because the tablet was not found. No further retry: Not found: Tablet not found: 71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:49.659359 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 (DEFAULT_TABLE table=default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.659879 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb62017acbd4472ac5b143eafb36ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.661211 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 717460af19164c058f42c7a03095bb30
I20240507 08:11:49.664357 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:49.664925 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 717460af19164c058f42c7a03095bb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 717460af19164c058f42c7a03095bb30 already in progress: creating tablet
I20240507 08:11:49.665938 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:49.666563 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 717460af19164c058f42c7a03095bb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 717460af19164c058f42c7a03095bb30 already in progress: creating tablet
I20240507 08:11:49.667260 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:49.668048 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 717460af19164c058f42c7a03095bb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 717460af19164c058f42c7a03095bb30 already in progress: creating tablet
W20240507 08:11:49.670214 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbf73c218b944679a12d769b1929e535 consensus state has no leader
I20240507 08:11:49.673976 5840 log.cc:824] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Log is configured to *not* fsync() on all Append() calls
I20240507 08:11:49.675089 5255 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet aeff693f1a214fdf97a3f8fc6ffec236
I20240507 08:11:49.672379 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf73c218b944679a12d769b1929e535. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.677325 5843 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.678236 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cb62017acbd4472ac5b143eafb36ae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.679283 5843 ts_tablet_manager.cc:1367] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.005s
I20240507 08:11:49.680779 5840 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:49.681051 5840 ts_tablet_manager.cc:1367] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.000s
I20240507 08:11:49.684585 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 81919c8f720f4c72bc2cc457b095859b (DEFAULT_TABLE table=default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.684995 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81919c8f720f4c72bc2cc457b095859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.686372 5694 tablet_service.cc:1467] Processing CreateTablet for tablet e8a046cce1f6467390071746a5e39fd9 (DEFAULT_TABLE table=default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.686745 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a046cce1f6467390071746a5e39fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.686866 5559 tablet_service.cc:1467] Processing CreateTablet for tablet e8a046cce1f6467390071746a5e39fd9 (DEFAULT_TABLE table=default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.687237 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a046cce1f6467390071746a5e39fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.688843 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 81919c8f720f4c72bc2cc457b095859b (DEFAULT_TABLE table=default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.689234 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81919c8f720f4c72bc2cc457b095859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.690058 5427 tablet_service.cc:1467] Processing CreateTablet for tablet e8a046cce1f6467390071746a5e39fd9 (DEFAULT_TABLE table=default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.690413 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8a046cce1f6467390071746a5e39fd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.693948 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 (DEFAULT_TABLE table=default.Table_2e8bf6d3e14e49419acb16babc07dd38 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.694375 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19efea8905cf4e83a523c278b2f16dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.698168 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 81919c8f720f4c72bc2cc457b095859b (DEFAULT_TABLE table=default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.698508 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81919c8f720f4c72bc2cc457b095859b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.699606 5843 raft_consensus.cc:348] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.701894 5843 raft_consensus.cc:374] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.702198 5843 raft_consensus.cc:729] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.703423 5843 consensus_queue.cc:260] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.706892 5839 tablet_bootstrap.cc:492] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:49.707227 5839 ts_tablet_manager.cc:1367] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.194s user 0.029s sys 0.051s
I20240507 08:11:49.708498 5695 tablet_service.cc:1467] Processing CreateTablet for tablet bc619db3a2944ecc9f0112b28de148a6 (DEFAULT_TABLE table=default.Table_19215ebf25be44c88d9a815d887298d2 [id=108648d0bbfe41cd961bdd8b7736989d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.708758 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 (DEFAULT_TABLE table=default.Table_2e8bf6d3e14e49419acb16babc07dd38 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.708911 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc619db3a2944ecc9f0112b28de148a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.709098 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19efea8905cf4e83a523c278b2f16dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.714824 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 (DEFAULT_TABLE table=default.Table_2e8bf6d3e14e49419acb16babc07dd38 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.715194 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19efea8905cf4e83a523c278b2f16dc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.716302 5843 ts_tablet_manager.cc:1398] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.037s user 0.006s sys 0.006s
I20240507 08:11:49.719260 5843 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.720077 5259 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a988072735474ed08bbd560ccf2ec169
I20240507 08:11:49.721031 5843 tablet_bootstrap.cc:654] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.723732 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.724990 5841 tablet_replica.cc:330] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:49.725387 5841 raft_consensus.cc:2229] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.725749 5841 raft_consensus.cc:2258] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:49.727406 5841 ts_tablet_manager.cc:1875] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:49.728543 5839 raft_consensus.cc:348] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.728963 5839 raft_consensus.cc:374] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.729038 5839 raft_consensus.cc:729] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.729622 5839 consensus_queue.cc:260] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.733873 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:49.734498 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cd6fa62dcd848118d45e77d25dfd6d4 already in progress: creating tablet
I20240507 08:11:49.734911 5841 ts_tablet_manager.cc:1888] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:49.735108 5841 log.cc:1196] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:49.735627 5841 ts_tablet_manager.cc:1909] T 0cd6fa62dcd848118d45e77d25dfd6d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:49.735751 5843 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.735911 5843 ts_tablet_manager.cc:1367] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20240507 08:11:49.737290 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (table default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]) successfully deleted
I20240507 08:11:49.737380 5839 ts_tablet_manager.cc:1398] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.030s user 0.000s sys 0.012s
I20240507 08:11:49.737821 5843 raft_consensus.cc:348] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.737990 5843 raft_consensus.cc:374] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.738106 5843 raft_consensus.cc:729] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.738334 5843 consensus_queue.cc:260] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.740867 5843 ts_tablet_manager.cc:1398] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:11:49.742725 5843 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.743402 5765 heartbeater.cc:503] Master 127.4.246.252:45297 was elected leader, sending a full tablet report...
I20240507 08:11:49.744577 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.746160 5843 tablet_bootstrap.cc:654] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.747714 5839 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:49.749336 5841 tablet_replica.cc:330] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:49.749491 5839 tablet_bootstrap.cc:654] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.752522 5841 raft_consensus.cc:2229] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.752768 5841 raft_consensus.cc:2258] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:49.755381 5840 raft_consensus.cc:348] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.756328 5840 raft_consensus.cc:374] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.756489 5840 raft_consensus.cc:729] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.757346 5840 consensus_queue.cc:260] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:49.759757 5633 tablet.cc:2369] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:11:49.760736 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c3dd460f347c47e9990f8cd2f4aeae71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.761471 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.767779 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet d6d9ab293113410c857b5de903e21eb5 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.769340 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ac1b4843bfc4cb491092f9b3204d8a4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.771634 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_03d2a58624f744daaa63f01a2b08b60e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.772377 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03d2a58624f744daaa63f01a2b08b60e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.779641 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f939facf624c472a893dccfe40ce5707"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.780282 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f939facf624c472a893dccfe40ce5707 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.781826 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1666cbd5c04f43f4aca54ecbed94ec7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.782434 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1666cbd5c04f43f4aca54ecbed94ec7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.786259 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5a63960776644f8186340cc40315f99f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.788293 5841 ts_tablet_manager.cc:1875] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:49.792408 5083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbFV97/build/asan/bin/kudu with pid 5159
I20240507 08:11:49.797693 5841 ts_tablet_manager.cc:1888] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:49.798883 5841 log.cc:1196] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:49.799078 5843 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.799271 5843 ts_tablet_manager.cc:1367] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.057s user 0.017s sys 0.009s
I20240507 08:11:49.799605 5841 ts_tablet_manager.cc:1909] T d6d9ab293113410c857b5de903e21eb5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:49.799789 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1ac1b4843bfc4cb491092f9b3204d8a4
I20240507 08:11:49.800247 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 1ac1b4843bfc4cb491092f9b3204d8a4 deleted and replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f
W20240507 08:11:49.800424 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a63960776644f8186340cc40315f99f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.800438 5843 raft_consensus.cc:348] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.800665 5843 raft_consensus.cc:374] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.800747 5843 raft_consensus.cc:729] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.801039 5843 consensus_queue.cc:260] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.801514 5843 ts_tablet_manager.cc:1398] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:49.801829 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d6d9ab293113410c857b5de903e21eb5 (table default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]) successfully deleted
I20240507 08:11:49.802079 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.802726 5841 tablet_replica.cc:330] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:49.802883 5841 raft_consensus.cc:2229] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.803051 5841 raft_consensus.cc:2258] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:49.803184 5840 ts_tablet_manager.cc:1398] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.122s user 0.022s sys 0.033s
I20240507 08:11:49.803483 5843 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.803656 5840 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:49.805130 5841 ts_tablet_manager.cc:1875] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:49.805271 5840 tablet_bootstrap.cc:654] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.806588 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.807602 5840 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:49.807739 5840 ts_tablet_manager.cc:1367] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
W20240507 08:11:49.808507 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac1b4843bfc4cb491092f9b3204d8a4 already in progress: creating tablet
I20240507 08:11:49.808604 5840 raft_consensus.cc:348] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.808776 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cb62017acbd4472ac5b143eafb36ae9 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.809106 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c3384c5c626c47158afbfaf8494a39ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.809392 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet e8a046cce1f6467390071746a5e39fd9 from {username='slave'} at 127.0.0.1:44738
W20240507 08:11:49.809618 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3384c5c626c47158afbfaf8494a39ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.810699 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.805174 5843 tablet_bootstrap.cc:654] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.811617 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet d6d9ab293113410c857b5de903e21eb5 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.808797 5840 raft_consensus.cc:374] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.812245 5840 raft_consensus.cc:729] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.812882 5840 consensus_queue.cc:260] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.813213 5841 ts_tablet_manager.cc:1888] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:49.813324 5841 log.cc:1196] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1ac1b4843bfc4cb491092f9b3204d8a4
I20240507 08:11:49.813676 5840 ts_tablet_manager.cc:1398] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240507 08:11:49.813724 5841 ts_tablet_manager.cc:1909] T 1ac1b4843bfc4cb491092f9b3204d8a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:49.813944 5840 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:49.814730 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:49.814893 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.815173 5838 tablet_replica.cc:330] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:49.815618 5838 raft_consensus.cc:2229] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.816016 5838 raft_consensus.cc:2258] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:49.817445 5840 tablet_bootstrap.cc:654] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.817891 5838 ts_tablet_manager.cc:1875] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:49.820566 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac1b4843bfc4cb491092f9b3204d8a4 already in progress: creating tablet
I20240507 08:11:49.820765 5843 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.820953 5843 ts_tablet_manager.cc:1367] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240507 08:11:49.821390 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_50c7cb69774c461e88d7610b086cc0ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.821923 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50c7cb69774c461e88d7610b086cc0ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.821983 5843 raft_consensus.cc:348] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.822235 5843 raft_consensus.cc:374] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.822335 5843 raft_consensus.cc:729] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.822599 5843 consensus_queue.cc:260] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.823275 5843 ts_tablet_manager.cc:1398] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:11:49.823634 5843 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.824339 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659" } new_table_name: "default.Table_01d80b8a1fde41f4938e1cdf5a191aa4" modify_external_catalogs: true
I20240507 08:11:49.825722 5843 tablet_bootstrap.cc:654] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.827142 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a5900476f17d41e883656a38ce31f881" } new_table_name: "default.Table_9152edd2e429414ea4670ea7d1ae0cc4" modify_external_catalogs: true
I20240507 08:11:49.834637 5843 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.834846 5843 ts_tablet_manager.cc:1367] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240507 08:11:49.835790 5843 raft_consensus.cc:348] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.836056 5843 raft_consensus.cc:374] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.836138 5843 raft_consensus.cc:729] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.836247 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0514ff21565c4cc394b734007163903e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.836829 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0514ff21565c4cc394b734007163903e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.836973 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1eb6ce65380844798cfe02364195022d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:49.837610 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eb6ce65380844798cfe02364195022d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.839066 5843 consensus_queue.cc:260] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.840525 5843 ts_tablet_manager.cc:1398] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240507 08:11:49.840935 5843 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.842021 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_19215ebf25be44c88d9a815d887298d2" } new_table_name: "default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7" modify_external_catalogs: true
I20240507 08:11:49.843132 5843 tablet_bootstrap.cc:654] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.843883 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.844059 5838 ts_tablet_manager.cc:1888] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:49.844213 5838 log.cc:1196] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:49.844261 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfa9a247eba402cb3f0b818ad9f837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.844663 5838 ts_tablet_manager.cc:1909] T 0cd6fa62dcd848118d45e77d25dfd6d4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:49.846966 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.847158 5839 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:49.847298 5839 ts_tablet_manager.cc:1367] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.100s user 0.012s sys 0.014s
I20240507 08:11:49.847334 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68b6c08f00d46779a31f4e9eed99f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.851562 5843 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.851742 5843 ts_tablet_manager.cc:1367] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240507 08:11:49.852692 5843 raft_consensus.cc:348] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.852995 5843 raft_consensus.cc:374] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.853120 5843 raft_consensus.cc:729] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.853487 5843 consensus_queue.cc:260] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.854408 5843 ts_tablet_manager.cc:1398] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:49.854736 5843 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.856798 5843 tablet_bootstrap.cc:654] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.858142 5562 tablet_service.cc:1467] Processing CreateTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.858530 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68b6c08f00d46779a31f4e9eed99f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.865123 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.865509 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.865939 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfa9a247eba402cb3f0b818ad9f837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.868456 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cfa9a247eba402cb3f0b818ad9f837a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.871716 5695 tablet_service.cc:1467] Processing CreateTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f (DEFAULT_TABLE table=default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.872092 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c68b6c08f00d46779a31f4e9eed99f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.875190 5839 raft_consensus.cc:348] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.875300 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (table default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]) successfully deleted
I20240507 08:11:49.875416 5839 raft_consensus.cc:374] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.875515 5839 raft_consensus.cc:729] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.878990 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a1e20af4278a4c35ab581d660cc9ec3e" } modify_external_catalogs: true
I20240507 08:11:49.879194 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a1e20af4278a4c35ab581d660cc9ec3e" } modify_external_catalogs: true
I20240507 08:11:49.879236 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_436c3bc45e4b4ab5a50defe8f2144a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:49.879725 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bd16315ec95647f699c27a2df5403d6e" } modify_external_catalogs: true
I20240507 08:11:49.879930 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bd16315ec95647f699c27a2df5403d6e" } modify_external_catalogs: true
W20240507 08:11:49.879747 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_436c3bc45e4b4ab5a50defe8f2144a08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:49.884790 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 1 got 0
I20240507 08:11:49.885725 5839 consensus_queue.cc:260] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.889459 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e8bf6d3e14e49419acb16babc07dd38" } new_table_name: "default.Table_a19ae1cf5dd64ba0bd8424ab78f287e4" modify_external_catalogs: true
I20240507 08:11:49.890059 5562 tablet_service.cc:1467] Processing CreateTablet for tablet baa842552d00489998f83e68aa0d516b (DEFAULT_TABLE table=default.Table_f939facf624c472a893dccfe40ce5707 [id=13e03112a7a641828153301cefc1774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.890435 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa842552d00489998f83e68aa0d516b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.893950 5839 ts_tablet_manager.cc:1398] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.047s user 0.001s sys 0.002s
I20240507 08:11:49.895694 5839 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:49.896579 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5cb62017acbd4472ac5b143eafb36ae9
I20240507 08:11:49.897135 5839 tablet_bootstrap.cc:654] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.897145 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 5cb62017acbd4472ac5b143eafb36ae9 deleted and replaced by tablet 1d16af3fd2a6448a82146edbb2362715
I20240507 08:11:49.897498 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_03d2a58624f744daaa63f01a2b08b60e" } modify_external_catalogs: true
I20240507 08:11:49.897794 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_03d2a58624f744daaa63f01a2b08b60e" } modify_external_catalogs: true
I20240507 08:11:49.898901 5700 tablet_service.cc:1467] Processing CreateTablet for tablet baa842552d00489998f83e68aa0d516b (DEFAULT_TABLE table=default.Table_f939facf624c472a893dccfe40ce5707 [id=13e03112a7a641828153301cefc1774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.899300 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa842552d00489998f83e68aa0d516b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.899883 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:49.900513 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5cb62017acbd4472ac5b143eafb36ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb62017acbd4472ac5b143eafb36ae9 already in progress: creating tablet
I20240507 08:11:49.900557 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.900954 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (DEFAULT_TABLE table=default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.901289 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.901885 5421 tablet_service.cc:1467] Processing CreateTablet for tablet baa842552d00489998f83e68aa0d516b (DEFAULT_TABLE table=default.Table_f939facf624c472a893dccfe40ce5707 [id=13e03112a7a641828153301cefc1774e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.902267 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet baa842552d00489998f83e68aa0d516b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.905879 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (DEFAULT_TABLE table=default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.906245 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.907497 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5cb62017acbd4472ac5b143eafb36ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb62017acbd4472ac5b143eafb36ae9 already in progress: creating tablet
I20240507 08:11:49.918134 5840 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:49.918385 5840 ts_tablet_manager.cc:1367] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.104s user 0.032s sys 0.064s
I20240507 08:11:49.918602 5839 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:49.918779 5839 ts_tablet_manager.cc:1367] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.002s
I20240507 08:11:49.919376 5840 raft_consensus.cc:348] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.919507 5839 raft_consensus.cc:348] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.919683 5840 raft_consensus.cc:374] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.919720 5839 raft_consensus.cc:374] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.919793 5839 raft_consensus.cc:729] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.919783 5840 raft_consensus.cc:729] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.920090 5839 consensus_queue.cc:260] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.920120 5840 consensus_queue.cc:260] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.920414 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.920871 5840 ts_tablet_manager.cc:1398] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 08:11:49.921252 5840 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:49.922673 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (DEFAULT_TABLE table=default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.923103 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.923411 5840 tablet_bootstrap.cc:654] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.923578 5839 ts_tablet_manager.cc:1398] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240507 08:11:49.925405 5839 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:49.930068 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5cb62017acbd4472ac5b143eafb36ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb62017acbd4472ac5b143eafb36ae9 already in progress: creating tablet
I20240507 08:11:49.934105 5839 tablet_bootstrap.cc:654] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.938892 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 120ebfd34a9e4d73a215cb9206013728 (DEFAULT_TABLE table=default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.939373 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120ebfd34a9e4d73a215cb9206013728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.940893 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:49.941447 5259 catalog_manager.cc:6554] ReplaceTablet: tablet e8a046cce1f6467390071746a5e39fd9 deleted and replaced by tablet 7a9edc839e3440809600f6128c649457
I20240507 08:11:49.941533 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet a485c1dc1cae40fcadce8ce75a5607b2 (table default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]). Expected version 1 got 0
I20240507 08:11:49.942183 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_3e9ec2d2986e4ec5a3dd46fdb9cbc659 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 1 got 0
I20240507 08:11:49.942448 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_a5900476f17d41e883656a38ce31f881 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 2 got 0
I20240507 08:11:49.942795 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9edc839e3440809600f6128c649457) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:49.943423 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e8a046cce1f6467390071746a5e39fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8a046cce1f6467390071746a5e39fd9 already in progress: creating tablet
W20240507 08:11:49.945219 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19efea8905cf4e83a523c278b2f16dc7 consensus state has no leader
I20240507 08:11:49.945375 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 120ebfd34a9e4d73a215cb9206013728 (DEFAULT_TABLE table=default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.945823 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120ebfd34a9e4d73a215cb9206013728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.945788 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f939facf624c472a893dccfe40ce5707" } new_table_name: "default.Table_b4b47cddd3c949bb9b91f6351807ceff" modify_external_catalogs: true
W20240507 08:11:49.947113 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc619db3a2944ecc9f0112b28de148a6 consensus state has no leader
I20240507 08:11:49.947546 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a873e078e67e44b4871044bbe3846aba" } modify_external_catalogs: true
I20240507 08:11:49.947783 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a873e078e67e44b4871044bbe3846aba" } modify_external_catalogs: true
W20240507 08:11:49.949894 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34ffe98d424a4eb5a08be067f96a50f9 consensus state has no leader
I20240507 08:11:49.950294 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_345d6952a3384ee29f957390161d28dc" } modify_external_catalogs: true
I20240507 08:11:49.950505 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_345d6952a3384ee29f957390161d28dc" } modify_external_catalogs: true
I20240507 08:11:49.944120 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9edc839e3440809600f6128c649457) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.952610 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 120ebfd34a9e4d73a215cb9206013728
W20240507 08:11:49.953267 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbf73c218b944679a12d769b1929e535 consensus state has no leader
W20240507 08:11:49.952613 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e8a046cce1f6467390071746a5e39fd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e8a046cce1f6467390071746a5e39fd9 already in progress: creating tablet
I20240507 08:11:49.953809 5839 tablet_bootstrap.cc:492] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:49.953958 5839 ts_tablet_manager.cc:1367] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.001s
I20240507 08:11:49.954058 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5cfe3c25a9a84eab9091502955453951
I20240507 08:11:49.954120 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.954262 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:49.954784 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 120ebfd34a9e4d73a215cb9206013728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ebfd34a9e4d73a215cb9206013728 already in progress: creating tablet
I20240507 08:11:49.954959 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.954862 5839 raft_consensus.cc:348] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.955070 5839 raft_consensus.cc:374] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:11:49.955183 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 120ebfd34a9e4d73a215cb9206013728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ebfd34a9e4d73a215cb9206013728 already in progress: creating tablet
I20240507 08:11:49.955209 5839 raft_consensus.cc:729] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.955333 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:49.955469 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5cfe3c25a9a84eab9091502955453951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cfe3c25a9a84eab9091502955453951 already in progress: creating tablet
W20240507 08:11:49.955864 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5cfe3c25a9a84eab9091502955453951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cfe3c25a9a84eab9091502955453951 already in progress: creating tablet
I20240507 08:11:49.955528 5839 consensus_queue.cc:260] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.956230 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.956866 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (replaced by tablet 7a9edc839e3440809600f6128c649457) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:49.958081 5838 tablet_replica.cc:330] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:49.958231 5843 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:49.958451 5843 ts_tablet_manager.cc:1367] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.032s
I20240507 08:11:49.958514 5838 raft_consensus.cc:2229] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.958884 5838 raft_consensus.cc:2258] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:49.959650 5838 ts_tablet_manager.cc:1875] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:49.959576 5843 raft_consensus.cc:348] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.959910 5843 raft_consensus.cc:374] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:49.959986 5843 raft_consensus.cc:729] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:49.960382 5843 consensus_queue.cc:260] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.961233 5843 ts_tablet_manager.cc:1398] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:11:49.961676 5843 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:49.963896 5843 tablet_bootstrap.cc:654] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.965886 5261 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7a9edc839e3440809600f6128c649457
I20240507 08:11:49.966208 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a485c1dc1cae40fcadce8ce75a5607b2
I20240507 08:11:49.966878 5264 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 1d16af3fd2a6448a82146edbb2362715
I20240507 08:11:49.967008 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.967424 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.967763 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 98880a5ff9604cb4981c033c99e80be7 (DEFAULT_TABLE table=default.Table_1666cbd5c04f43f4aca54ecbed94ec7a [id=245f6e995eca47f38bdc7495b40a6299]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:49.968099 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a485c1dc1cae40fcadce8ce75a5607b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485c1dc1cae40fcadce8ce75a5607b2 already in progress: creating tablet
I20240507 08:11:49.968158 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98880a5ff9604cb4981c033c99e80be7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:49.968645 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet baa842552d00489998f83e68aa0d516b consensus state has no leader
W20240507 08:11:49.969882 5768 tablet.cc:2369] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:11:49.971499 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 120ebfd34a9e4d73a215cb9206013728 (DEFAULT_TABLE table=default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.972133 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120ebfd34a9e4d73a215cb9206013728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.973856 5839 ts_tablet_manager.cc:1398] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240507 08:11:49.975448 5851 raft_consensus.cc:482] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:49.975697 5851 raft_consensus.cc:504] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:49.977797 5851 leader_election.cc:288] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:49.977952 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 98880a5ff9604cb4981c033c99e80be7 (DEFAULT_TABLE table=default.Table_1666cbd5c04f43f4aca54ecbed94ec7a [id=245f6e995eca47f38bdc7495b40a6299]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:49.978403 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98880a5ff9604cb4981c033c99e80be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:49.986996 5839 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:49.988794 5839 tablet_bootstrap.cc:654] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:49.989923 5838 ts_tablet_manager.cc:1888] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:49.990082 5838 log.cc:1196] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:49.990216 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.990460 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:49.991062 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cb62017acbd4472ac5b143eafb36ae9 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:49.990466 5838 ts_tablet_manager.cc:1909] T e4a66fa7446a4dbea3885f8778e7f92b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:49.994117 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.994277 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e4a66fa7446a4dbea3885f8778e7f92b (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
I20240507 08:11:49.994426 5841 tablet_replica.cc:330] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:49.994650 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.994661 5841 raft_consensus.cc:2229] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.994886 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:49.994906 5841 raft_consensus.cc:2258] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:49.995062 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a485c1dc1cae40fcadce8ce75a5607b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485c1dc1cae40fcadce8ce75a5607b2 already in progress: creating tablet
I20240507 08:11:49.995414 5838 tablet_replica.cc:330] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:49.995553 5841 ts_tablet_manager.cc:1875] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:49.995575 5838 raft_consensus.cc:2229] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:49.995754 5838 raft_consensus.cc:2258] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:49.996069 5655 connection.cc:535] client connection to 127.4.246.253:43459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 08:11:49.996544 5767 heartbeater.cc:644] Failed to heartbeat to 127.4.246.253:43459 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 08:11:49.997098 5632 heartbeater.cc:644] Failed to heartbeat to 127.4.246.253:43459 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.4.246.253:43459 (error 108)
I20240507 08:11:49.998217 5846 raft_consensus.cc:482] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:49.998471 5846 raft_consensus.cc:504] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.000707 5846 leader_election.cc:288] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.001961 5841 ts_tablet_manager.cc:1888] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.002151 5841 log.cc:1196] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/86b8cfe64e5f418bbfb65087003beb66
I20240507 08:11:50.002719 5841 ts_tablet_manager.cc:1909] T 86b8cfe64e5f418bbfb65087003beb66 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:11:50.003732 5495 tablet.cc:2369] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:11:50.004053 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 86b8cfe64e5f418bbfb65087003beb66 (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:50.004455 5841 tablet_replica.cc:330] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:11:50.004585 5379 connection.cc:535] client connection to 127.4.246.253:43459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 08:11:50.005093 5494 heartbeater.cc:644] Failed to heartbeat to 127.4.246.253:43459 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240507 08:11:50.007243 5841 raft_consensus.cc:2229] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.007475 5841 raft_consensus.cc:2258] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.009270 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece (DEFAULT_TABLE table=default.Table_436c3bc45e4b4ab5a50defe8f2144a08 [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.010258 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708d4bee30cc45bd9b86c56beb2c3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.011906 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 (DEFAULT_TABLE table=default.Table_c3384c5c626c47158afbfaf8494a39ff [id=a2716884f32e46468e513117cd78b7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.012373 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5220905954dc9b078e00e0db69fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.013942 5423 tablet_service.cc:1467] Processing CreateTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 (DEFAULT_TABLE table=default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.014369 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1063aa5c7014d84b66f74ce28e3d890. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.014349 5250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.246.253:43459 (error 108)
I20240507 08:11:50.017917 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab (DEFAULT_TABLE table=default.Table_0514ff21565c4cc394b734007163903e [id=079eefc78b9445f098838f484af7a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.018339 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400d2a7b0e224e46a5a580ca22e96eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.020267 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 1ac1b4843bfc4cb491092f9b3204d8a4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.022457 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9d3d56b2117448afac241847a046a08a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.023170 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d3d56b2117448afac241847a046a08a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.029629 5838 ts_tablet_manager.cc:1875] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.032763 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_11230a32b0464b20850876ba4b65666e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.033336 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11230a32b0464b20850876ba4b65666e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.034058 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 71bbe28a013344bcb6529247d7139ab9 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.034274 5420 tablet_service.cc:1467] Processing CreateTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 (DEFAULT_TABLE table=default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.034739 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0074cc70e248ac8b5e90cf57b441c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.043532 5841 ts_tablet_manager.cc:1875] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.044018 5840 tablet_bootstrap.cc:492] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.044247 5840 ts_tablet_manager.cc:1367] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.123s user 0.022s sys 0.027s
I20240507 08:11:50.045193 5839 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.045151 5840 raft_consensus.cc:348] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.045379 5840 raft_consensus.cc:374] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.045475 5840 raft_consensus.cc:729] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.045756 5840 consensus_queue.cc:260] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.046422 5840 ts_tablet_manager.cc:1398] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:50.046543 5839 ts_tablet_manager.cc:1367] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20240507 08:11:50.046720 5840 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.047245 5839 raft_consensus.cc:348] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.047502 5839 raft_consensus.cc:374] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.047590 5839 raft_consensus.cc:729] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.050117 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 87e1f43426b14f98b985656f8d0d7141 (DEFAULT_TABLE table=default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.052103 5839 consensus_queue.cc:260] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:50.053974 5863 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 08:11:49.997616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.246.253:43459 (local address 127.4.246.195:49375)
0507 08:11:50.052201 (+ 54585us) negotiation.cc:107] Waiting for socket to connect
0507 08:11:50.052379 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":54471,"thread_start_us":54192,"threads_started":1}
I20240507 08:11:50.054016 5840 tablet_bootstrap.cc:654] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.055580 5843 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.055796 5843 ts_tablet_manager.cc:1367] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.001s
I20240507 08:11:50.056607 5843 raft_consensus.cc:348] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.056864 5843 raft_consensus.cc:374] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.057008 5843 raft_consensus.cc:729] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.057276 5843 consensus_queue.cc:260] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.058203 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.059083 5843 ts_tablet_manager.cc:1398] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:50.059337 5843 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.060448 5851 raft_consensus.cc:482] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.060497 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.060607 5851 raft_consensus.cc:504] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.061046 5843 tablet_bootstrap.cc:654] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.061228 5851 leader_election.cc:288] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:50.061862 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.062574 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.063227 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 0cd6fa62dcd848118d45e77d25dfd6d4 with delete_type TABLET_DATA_DELETED (replaced by tablet 34ffe98d424a4eb5a08be067f96a50f9) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.063534 5837 tablet_replica.cc:330] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.063735 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 87ede5b03251463d8608d15f037e4484 (DEFAULT_TABLE table=default.Table_11230a32b0464b20850876ba4b65666e [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.064139 5837 raft_consensus.cc:2229] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.064581 5837 raft_consensus.cc:2258] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.065327 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f4619389ae4548d3add23a0eaf7bc6aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.066032 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4619389ae4548d3add23a0eaf7bc6aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.066458 5837 ts_tablet_manager.cc:1875] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.069972 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece (DEFAULT_TABLE table=default.Table_436c3bc45e4b4ab5a50defe8f2144a08 [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.070392 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708d4bee30cc45bd9b86c56beb2c3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.073894 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.074285 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.074653 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.074999 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 9922d86933cc40dda1efff9ef03dcdf2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.075322 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet e4a66fa7446a4dbea3885f8778e7f92b with delete_type TABLET_DATA_DELETED (replaced by tablet 5cfe3c25a9a84eab9091502955453951) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.075672 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.076009 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.076320 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.076625 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.076545 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a51f156faabe436288474d53bdb248ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.077153 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 98880a5ff9604cb4981c033c99e80be7 (DEFAULT_TABLE table=default.Table_1666cbd5c04f43f4aca54ecbed94ec7a [id=245f6e995eca47f38bdc7495b40a6299]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.077250 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a51f156faabe436288474d53bdb248ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.077593 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98880a5ff9604cb4981c033c99e80be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.078413 5560 tablet_service.cc:1467] Processing CreateTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 (DEFAULT_TABLE table=default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.078758 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1063aa5c7014d84b66f74ce28e3d890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.079367 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_66645dce4ba84479878b057a6178138a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.079825 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66645dce4ba84479878b057a6178138a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.083381 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a477a636c30a4d289f6c00d2975f1e5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.084045 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a477a636c30a4d289f6c00d2975f1e5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.084333 5698 tablet_service.cc:1467] Processing CreateTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 (DEFAULT_TABLE table=default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.084676 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0074cc70e248ac8b5e90cf57b441c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.084980 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 (DEFAULT_TABLE table=default.Table_c3384c5c626c47158afbfaf8494a39ff [id=a2716884f32e46468e513117cd78b7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.085143 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab (DEFAULT_TABLE table=default.Table_0514ff21565c4cc394b734007163903e [id=079eefc78b9445f098838f484af7a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.085350 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5220905954dc9b078e00e0db69fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.085521 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400d2a7b0e224e46a5a580ca22e96eab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.085963 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab (DEFAULT_TABLE table=default.Table_0514ff21565c4cc394b734007163903e [id=079eefc78b9445f098838f484af7a3ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.085963 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8b96df6db03e4f77a97662d764097c4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.086289 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400d2a7b0e224e46a5a580ca22e96eab. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.086405 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b96df6db03e4f77a97662d764097c4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.087499 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 (DEFAULT_TABLE table=default.Table_c3384c5c626c47158afbfaf8494a39ff [id=a2716884f32e46468e513117cd78b7af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.087662 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8aaa2c0a5ef44d2bb4718bbc70104607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.087837 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ea5220905954dc9b078e00e0db69fd2. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.088094 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.089341 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e3e2810edd06425882e48b9aeaada4f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.089780 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e2810edd06425882e48b9aeaada4f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.089846 5561 tablet_service.cc:1467] Processing CreateTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 (DEFAULT_TABLE table=default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.090167 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0074cc70e248ac8b5e90cf57b441c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.091545 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece (DEFAULT_TABLE table=default.Table_436c3bc45e4b4ab5a50defe8f2144a08 [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.092000 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 708d4bee30cc45bd9b86c56beb2c3ece. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.093817 5700 tablet_service.cc:1467] Processing CreateTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 (DEFAULT_TABLE table=default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.094170 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1063aa5c7014d84b66f74ce28e3d890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.096119 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 87e1f43426b14f98b985656f8d0d7141 (DEFAULT_TABLE table=default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.096292 5839 ts_tablet_manager.cc:1398] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
I20240507 08:11:50.096474 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e1f43426b14f98b985656f8d0d7141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.098074 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 87ede5b03251463d8608d15f037e4484 (DEFAULT_TABLE table=default.Table_11230a32b0464b20850876ba4b65666e [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.098412 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ede5b03251463d8608d15f037e4484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.098745 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e1f43426b14f98b985656f8d0d7141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.099259 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ede5b03251463d8608d15f037e4484. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.103924 5839 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.106678 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 (DEFAULT_TABLE table=default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.107167 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f6f62c56cb4c29bf05914ca461a559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.114595 5841 ts_tablet_manager.cc:1888] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.121219 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 87ede5b03251463d8608d15f037e4484 (DEFAULT_TABLE table=default.Table_11230a32b0464b20850876ba4b65666e [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.121873 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:50.127041 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 (DEFAULT_TABLE table=default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.127178 5841 log.cc:1196] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:50.127460 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f6f62c56cb4c29bf05914ca461a559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.127575 5841 ts_tablet_manager.cc:1909] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.127790 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_97e6dd7c062b4dab94e874be91f39e25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.127853 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 87e1f43426b14f98b985656f8d0d7141 (DEFAULT_TABLE table=default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.128208 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87ede5b03251463d8608d15f037e4484. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.128327 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97e6dd7c062b4dab94e874be91f39e25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.129871 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87e1f43426b14f98b985656f8d0d7141. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.129998 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ede5b03251463d8608d15f037e4484 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.130508 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e4a66fa7446a4dbea3885f8778e7f92b (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
W20240507 08:11:50.116369 5868 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 08:11:50.005616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.246.253:43459 (local address 127.4.246.193:41233)
0507 08:11:50.115042 (+109426us) negotiation.cc:107] Waiting for socket to connect
0507 08:11:50.115290 (+ 248us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":109303,"thread_start_us":35989,"threads_started":1}
I20240507 08:11:50.131466 5564 tablet_service.cc:1467] Processing CreateTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb (DEFAULT_TABLE table=default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.131865 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b8044e4a0a41218f67513b14c0ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.132223 5841 tablet_replica.cc:330] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.132396 5841 raft_consensus.cc:2229] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.132541 5841 raft_consensus.cc:2258] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:50.134370 5866 negotiation.cc:337] Failed RPC negotiation. Trace:
0507 08:11:49.997615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.246.253:43459 (local address 127.4.246.194:56427)
0507 08:11:50.128888 (+131273us) negotiation.cc:107] Waiting for socket to connect
0507 08:11:50.129105 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":131132,"threads_started":1}
I20240507 08:11:50.136180 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 34e4e33f52df432b8d043171d7232c2a (DEFAULT_TABLE table=default.Table_66645dce4ba84479878b057a6178138a [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.136577 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e4e33f52df432b8d043171d7232c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.138737 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:50.142328 5841 ts_tablet_manager.cc:1875] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.145849 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:50.146170 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
I20240507 08:11:50.150594 5843 tablet_bootstrap.cc:492] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.150766 5843 ts_tablet_manager.cc:1367] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.013s
I20240507 08:11:50.151575 5843 raft_consensus.cc:348] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.151798 5843 raft_consensus.cc:374] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.151727 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb (DEFAULT_TABLE table=default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.151893 5843 raft_consensus.cc:729] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:11:50.152700 5656 leader_election.cc:341] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.152915 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b8044e4a0a41218f67513b14c0ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.152801 5843 consensus_queue.cc:260] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:50.153075 5656 leader_election.cc:341] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:50.153353 5837 ts_tablet_manager.cc:1888] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.153481 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 34e4e33f52df432b8d043171d7232c2a (DEFAULT_TABLE table=default.Table_66645dce4ba84479878b057a6178138a [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.153928 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e4e33f52df432b8d043171d7232c2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.156419 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.157742 5843 ts_tablet_manager.cc:1398] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240507 08:11:50.161345 5843 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.164044 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 19efea8905cf4e83a523c278b2f16dc7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.164371 5843 tablet_bootstrap.cc:654] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.164486 5837 log.cc:1196] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:50.165467 5837 ts_tablet_manager.cc:1909] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.166268 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet aa1942369ba1413cb1c1198d4c972234 with delete_type TABLET_DATA_DELETED (replaced by tablet cbf73c218b944679a12d769b1929e535) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.168540 5837 tablet_replica.cc:330] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.168845 5837 raft_consensus.cc:2229] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.169018 5837 raft_consensus.cc:2258] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.170483 5843 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.170722 5837 ts_tablet_manager.cc:1875] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.171062 5843 ts_tablet_manager.cc:1367] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240507 08:11:50.171103 5848 raft_consensus.cc:482] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.171357 5848 raft_consensus.cc:504] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.171803 5843 raft_consensus.cc:348] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.172039 5843 raft_consensus.cc:374] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.172127 5843 raft_consensus.cc:729] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.173358 5848 leader_election.cc:288] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.176586 5841 ts_tablet_manager.cc:1888] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.176714 5841 log.cc:1196] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/9922d86933cc40dda1efff9ef03dcdf2
I20240507 08:11:50.177146 5841 ts_tablet_manager.cc:1909] T 9922d86933cc40dda1efff9ef03dcdf2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.178133 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 9922d86933cc40dda1efff9ef03dcdf2 (table default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]) successfully deleted
I20240507 08:11:50.178751 5841 tablet_replica.cc:330] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.178975 5841 raft_consensus.cc:2229] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.179229 5841 raft_consensus.cc:2258] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.179463 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_436c3bc45e4b4ab5a50defe8f2144a08" } new_table_name: "default.Table_745f979f496a402fb4868ac8ed7099ff" modify_external_catalogs: true
I20240507 08:11:50.179913 5841 ts_tablet_manager.cc:1875] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.180632 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_97e6dd7c062b4dab94e874be91f39e25" } new_table_name: "default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1" modify_external_catalogs: true
I20240507 08:11:50.181026 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_01d80b8a1fde41f4938e1cdf5a191aa4" } new_table_name: "default.Table_dec664d2f47d45eb924462d597822a47" modify_external_catalogs: true
I20240507 08:11:50.182194 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:50.182432 5447 raft_consensus.cc:2454] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
I20240507 08:11:50.182412 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4619389ae4548d3add23a0eaf7bc6aa" } modify_external_catalogs: true
I20240507 08:11:50.182586 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4619389ae4548d3add23a0eaf7bc6aa" } modify_external_catalogs: true
I20240507 08:11:50.182920 5520 leader_election.cc:302] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:11:50.183370 5848 raft_consensus.cc:2781] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.183511 5848 raft_consensus.cc:482] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.183598 5848 raft_consensus.cc:3037] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.186462 5848 raft_consensus.cc:504] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.186764 5841 ts_tablet_manager.cc:1888] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.186887 5841 log.cc:1196] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/717460af19164c058f42c7a03095bb30
I20240507 08:11:50.187247 5841 ts_tablet_manager.cc:1909] T 717460af19164c058f42c7a03095bb30 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.187584 5848 leader_election.cc:288] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.188233 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:50.188277 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 717460af19164c058f42c7a03095bb30 (table default.Table_8051479eeaf74dd8beccf13762a64249 [id=5d190d55ad4748cca262a1a33e12b68a]) successfully deleted
I20240507 08:11:50.188448 5447 raft_consensus.cc:3037] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.188599 5841 tablet_replica.cc:330] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.173645 5843 consensus_queue.cc:260] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.190447 5843 ts_tablet_manager.cc:1398] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240507 08:11:50.191710 5843 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.192167 5447 raft_consensus.cc:2454] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:11:50.194615 5843 tablet_bootstrap.cc:654] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.195997 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8da8971bb4e54275af1bd500580ef3c4" } modify_external_catalogs: true
I20240507 08:11:50.196168 5520 leader_election.cc:302] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:11:50.196238 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:50.196246 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8da8971bb4e54275af1bd500580ef3c4" } modify_external_catalogs: true
I20240507 08:11:50.196487 5848 raft_consensus.cc:2781] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.196537 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9d3d56b2117448afac241847a046a08a" } modify_external_catalogs: true
I20240507 08:11:50.196708 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9d3d56b2117448afac241847a046a08a" } modify_external_catalogs: true
I20240507 08:11:50.197065 5841 raft_consensus.cc:2229] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.197006 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_11230a32b0464b20850876ba4b65666e" } new_table_name: "default.Table_25edacce85f54417862546993cc8d161" modify_external_catalogs: true
I20240507 08:11:50.198163 5841 raft_consensus.cc:2258] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.198351 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a51f156faabe436288474d53bdb248ad" } modify_external_catalogs: true
I20240507 08:11:50.198500 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a51f156faabe436288474d53bdb248ad" } modify_external_catalogs: true
I20240507 08:11:50.198925 5841 ts_tablet_manager.cc:1875] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.206418 5838 ts_tablet_manager.cc:1888] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.206609 5838 log.cc:1196] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/9922d86933cc40dda1efff9ef03dcdf2
I20240507 08:11:50.207113 5838 ts_tablet_manager.cc:1909] T 9922d86933cc40dda1efff9ef03dcdf2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.208071 5838 tablet_replica.cc:330] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:11:50.210312 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 120ebfd34a9e4d73a215cb9206013728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120ebfd34a9e4d73a215cb9206013728 already in progress: creating tablet
W20240507 08:11:50.210686 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5cfe3c25a9a84eab9091502955453951 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cfe3c25a9a84eab9091502955453951 already in progress: creating tablet
I20240507 08:11:50.211124 5699 tablet_service.cc:1467] Processing CreateTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb (DEFAULT_TABLE table=default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.211422 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a485c1dc1cae40fcadce8ce75a5607b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a485c1dc1cae40fcadce8ce75a5607b2 already in progress: creating tablet
I20240507 08:11:50.211535 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9b8044e4a0a41218f67513b14c0ebeb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.212095 5654 leader_election.cc:341] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: 0cd6fa62dcd848118d45e77d25dfd6d4
I20240507 08:11:50.212234 5654 leader_election.cc:302] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.212478 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 0cd6fa62dcd848118d45e77d25dfd6d4 (table default.Table_01d80b8a1fde41f4938e1cdf5a191aa4 [id=98dc15c6d4b54ec2a8c55a0499349213]) successfully deleted
W20240507 08:11:50.212610 5654 raft_consensus.cc:2670] T 0cd6fa62dcd848118d45e77d25dfd6d4 P 1cbb887175f345ada2433adaec906fd4: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:11:50.212759 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ac1b4843bfc4cb491092f9b3204d8a4 already in progress: creating tablet
W20240507 08:11:50.212987 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 71bbe28a013344bcb6529247d7139ab9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71bbe28a013344bcb6529247d7139ab9 already in progress: creating tablet
W20240507 08:11:50.213200 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 86b8cfe64e5f418bbfb65087003beb66 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86b8cfe64e5f418bbfb65087003beb66 already in progress: creating tablet
I20240507 08:11:50.213747 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 (DEFAULT_TABLE table=default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.214696 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f6f62c56cb4c29bf05914ca461a559. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.220410 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 87ede5b03251463d8608d15f037e4484
I20240507 08:11:50.220877 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 87ede5b03251463d8608d15f037e4484 deleted and replaced by tablet 91c6cb7920df47aa97be6555c587798a
W20240507 08:11:50.220713 5378 leader_election.cc:341] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:50.221966 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 34e4e33f52df432b8d043171d7232c2a (DEFAULT_TABLE table=default.Table_66645dce4ba84479878b057a6178138a [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.223763 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 708d4bee30cc45bd9b86c56beb2c3ece consensus state has no leader
I20240507 08:11:50.224786 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34e4e33f52df432b8d043171d7232c2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.225049 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a6a35991c18414b9bc714e9a0ba14d8 has no consensus state
I20240507 08:11:50.226580 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:50.227223 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 87e1f43426b14f98b985656f8d0d7141 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.227228 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.227720 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_50c7cb69774c461e88d7610b086cc0ba" } modify_external_catalogs: true
I20240507 08:11:50.227908 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_50c7cb69774c461e88d7610b086cc0ba" } modify_external_catalogs: true
I20240507 08:11:50.229146 5841 ts_tablet_manager.cc:1888] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.229295 5841 log.cc:1196] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/aa1942369ba1413cb1c1198d4c972234
I20240507 08:11:50.229307 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c3384c5c626c47158afbfaf8494a39ff" } new_table_name: "default.Table_b650283bfd9b448c8a0ba81f514f2831" modify_external_catalogs: true
W20240507 08:11:50.231374 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34ffe98d424a4eb5a08be067f96a50f9 consensus state has no leader
I20240507 08:11:50.232609 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c68b6c08f00d46779a31f4e9eed99f9f
I20240507 08:11:50.233748 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.234465 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.235287 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.235401 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_169e8150dda24f87ac304a6c9031909d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.235700 5838 raft_consensus.cc:2229] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.235877 5838 raft_consensus.cc:2258] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:50.235932 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_169e8150dda24f87ac304a6c9031909d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.211504 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.237725 5843 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.237905 5843 ts_tablet_manager.cc:1367] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.011s
I20240507 08:11:50.238335 5838 ts_tablet_manager.cc:1875] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.238655 5843 raft_consensus.cc:348] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.238895 5843 raft_consensus.cc:374] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.238972 5843 raft_consensus.cc:729] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.219787 5839 tablet_bootstrap.cc:654] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.239192 5843 consensus_queue.cc:260] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.230605 5848 raft_consensus.cc:686] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:11:50.242172 5848 consensus_queue.cc:237] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.243741 5843 ts_tablet_manager.cc:1398] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:50.247232 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.247440 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.248301 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 9922d86933cc40dda1efff9ef03dcdf2 (table default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]) successfully deleted
I20240507 08:11:50.249794 5840 tablet_bootstrap.cc:492] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.249997 5840 ts_tablet_manager.cc:1367] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.203s user 0.017s sys 0.012s
I20240507 08:11:50.250761 5841 ts_tablet_manager.cc:1909] T aa1942369ba1413cb1c1198d4c972234 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.250773 5840 raft_consensus.cc:348] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.250993 5840 raft_consensus.cc:374] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.251068 5840 raft_consensus.cc:729] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.251322 5840 consensus_queue.cc:260] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.251694 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet aa1942369ba1413cb1c1198d4c972234 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
W20240507 08:11:50.252460 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 87ede5b03251463d8608d15f037e4484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ede5b03251463d8608d15f037e4484 already in progress: creating tablet
W20240507 08:11:50.252715 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c68b6c08f00d46779a31f4e9eed99f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c68b6c08f00d46779a31f4e9eed99f9f already in progress: creating tablet
I20240507 08:11:50.254325 5840 ts_tablet_manager.cc:1398] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20240507 08:11:50.254495 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91c6cb7920df47aa97be6555c587798a has no consensus state
I20240507 08:11:50.256299 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c9b8044e4a0a41218f67513b14c0ebeb
I20240507 08:11:50.256394 5840 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.257704 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.259285 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.259126 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.259605 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:50.259845 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.260484 5840 tablet_bootstrap.cc:654] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.259649 5843 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.262370 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.263155 5843 tablet_bootstrap.cc:654] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.263955 5841 tablet_replica.cc:330] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.264118 5841 raft_consensus.cc:2229] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.264286 5841 raft_consensus.cc:2258] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.265273 5841 ts_tablet_manager.cc:1875] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.266377 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 86b8cfe64e5f418bbfb65087003beb66 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ac1b4843bfc4cb491092f9b3204d8a4) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.267613 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ed0074cc70e248ac8b5e90cf57b441c3
I20240507 08:11:50.268359 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.269212 5840 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.269364 5840 ts_tablet_manager.cc:1367] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240507 08:11:50.270006 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.271041 5840 raft_consensus.cc:348] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.271245 5840 raft_consensus.cc:374] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.271343 5840 raft_consensus.cc:729] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.272334 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.272675 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.272830 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa1942369ba1413cb1c1198d4c972234" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:50.273137 5720 raft_consensus.cc:2454] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
W20240507 08:11:50.284768 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ea5220905954dc9b078e00e0db69fd2 consensus state has no leader
I20240507 08:11:50.285890 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fcab59c9e0064c3487c28efc45b90928"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.286450 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcab59c9e0064c3487c28efc45b90928 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.289155 5851 raft_consensus.cc:482] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.289325 5851 raft_consensus.cc:504] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.290103 5851 leader_election.cc:288] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:11:50.291494 5656 leader_election.cc:341] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:50.292531 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4a66fa7446a4dbea3885f8778e7f92b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.292742 5719 raft_consensus.cc:2454] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:50.293300 5377 leader_election.cc:302] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:50.293717 5377 raft_consensus.cc:2670] T e4a66fa7446a4dbea3885f8778e7f92b P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:50.295099 5837 ts_tablet_manager.cc:1888] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.300859 5838 ts_tablet_manager.cc:1888] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.301409 5841 ts_tablet_manager.cc:1888] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.301921 5841 log.cc:1196] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5cfe3c25a9a84eab9091502955453951
I20240507 08:11:50.302341 5841 ts_tablet_manager.cc:1909] T 5cfe3c25a9a84eab9091502955453951 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.302953 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_55f22a879f524a07ba3a6edf9b32d822"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.303506 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55f22a879f524a07ba3a6edf9b32d822 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.304616 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5cfe3c25a9a84eab9091502955453951 (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
W20240507 08:11:50.304888 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c9b8044e4a0a41218f67513b14c0ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b8044e4a0a41218f67513b14c0ebeb already in progress: creating tablet
I20240507 08:11:50.304831 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_678c1601578c46188487c197115995aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.305104 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ed0074cc70e248ac8b5e90cf57b441c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0074cc70e248ac8b5e90cf57b441c3 already in progress: creating tablet
W20240507 08:11:50.305281 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_678c1601578c46188487c197115995aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.306396 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet a485c1dc1cae40fcadce8ce75a5607b2 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.310775 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 717460af19164c058f42c7a03095bb30 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.311002 5838 log.cc:1196] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/86b8cfe64e5f418bbfb65087003beb66
I20240507 08:11:50.311342 5838 ts_tablet_manager.cc:1909] T 86b8cfe64e5f418bbfb65087003beb66 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.312178 5838 tablet_replica.cc:330] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.312333 5838 raft_consensus.cc:2229] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:50.313238 5420 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 (DEFAULT_TABLE table=default.Table_8b96df6db03e4f77a97662d764097c4c [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.313359 5838 pending_rounds.cc:62] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Trying to abort 1 pending ops.
I20240507 08:11:50.314342 5422 tablet_service.cc:1467] Processing CreateTablet for tablet fa932c6ad48f4ac183aabba8699cb148 (DEFAULT_TABLE table=default.Table_e3e2810edd06425882e48b9aeaada4f6 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.313496 5838 pending_rounds.cc:68] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924713960521728 op_type: NO_OP noop_request { }
I20240507 08:11:50.314702 5838 raft_consensus.cc:2866] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:50.314846 5838 raft_consensus.cc:2258] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.314846 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d (DEFAULT_TABLE table=default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.315137 5887 consensus_queue.cc:1159] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fe673739d8084e7da5d7edd722e857d8" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240507 08:11:50.314496 5839 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.315505 5839 ts_tablet_manager.cc:1367] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.212s user 0.008s sys 0.014s
I20240507 08:11:50.315650 5421 tablet_service.cc:1467] Processing CreateTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 (DEFAULT_TABLE table=default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.315979 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf2df0d3bd46268e22713fcc135a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.316248 5839 raft_consensus.cc:348] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.316435 5839 raft_consensus.cc:374] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.316517 5839 raft_consensus.cc:729] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.316771 5839 consensus_queue.cc:260] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.317270 5839 ts_tablet_manager.cc:1398] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:11:50.317610 5839 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.319525 5839 tablet_bootstrap.cc:654] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.320873 5838 ts_tablet_manager.cc:1875] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.320991 5564 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 (DEFAULT_TABLE table=default.Table_8b96df6db03e4f77a97662d764097c4c [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.321432 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf2df0d3bd46268e22713fcc135a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.321969 5565 tablet_service.cc:1467] Processing CreateTablet for tablet fa932c6ad48f4ac183aabba8699cb148 (DEFAULT_TABLE table=default.Table_e3e2810edd06425882e48b9aeaada4f6 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.321990 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9aa4c69483c04eb1a1c0db1c0555cef1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.322373 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa932c6ad48f4ac183aabba8699cb148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.322491 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f968f122c39b4fd5b995e8a2848a52d4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.322557 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.323024 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d (DEFAULT_TABLE table=default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.323360 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09adb78ad08a4e759eb9344cbabb6a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.325332 5562 tablet_service.cc:1467] Processing CreateTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 (DEFAULT_TABLE table=default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.325716 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09adb78ad08a4e759eb9344cbabb6a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.326741 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f968f122c39b4fd5b995e8a2848a52d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.329936 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa932c6ad48f4ac183aabba8699cb148. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.332464 5838 ts_tablet_manager.cc:1888] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:50.334138 5838 log.cc:1196] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/aa1942369ba1413cb1c1198d4c972234
I20240507 08:11:50.334851 5838 ts_tablet_manager.cc:1909] T aa1942369ba1413cb1c1198d4c972234 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.335275 5837 log.cc:1196] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/9922d86933cc40dda1efff9ef03dcdf2
I20240507 08:11:50.335786 5837 ts_tablet_manager.cc:1909] T 9922d86933cc40dda1efff9ef03dcdf2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.336182 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cdb3e713b1194129bd3f5be0ac3991e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.336436 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet aa1942369ba1413cb1c1198d4c972234 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
W20240507 08:11:50.336757 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdb3e713b1194129bd3f5be0ac3991e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:50.338212 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 87ede5b03251463d8608d15f037e4484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ede5b03251463d8608d15f037e4484 already in progress: creating tablet
I20240507 08:11:50.338166 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a5f462941b57438eb01a3cc87a0390bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.338903 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5f462941b57438eb01a3cc87a0390bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:50.339064 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c68b6c08f00d46779a31f4e9eed99f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c68b6c08f00d46779a31f4e9eed99f9f already in progress: creating tablet
W20240507 08:11:50.339838 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c9b8044e4a0a41218f67513b14c0ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b8044e4a0a41218f67513b14c0ebeb already in progress: creating tablet
W20240507 08:11:50.340601 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ed0074cc70e248ac8b5e90cf57b441c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0074cc70e248ac8b5e90cf57b441c3 already in progress: creating tablet
I20240507 08:11:50.342401 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4e6dbc6b805d4d6888de30dc35f6972b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.342885 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e6dbc6b805d4d6888de30dc35f6972b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.346343 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.346663 5838 tablet_replica.cc:330] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.346880 5838 raft_consensus.cc:2229] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.347121 5838 raft_consensus.cc:2258] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:50.301743 5654 leader_election.cc:341] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:50.347568 5654 leader_election.cc:302] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:50.347934 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 46207.9us
I20240507 08:11:50.348472 5843 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.348680 5843 ts_tablet_manager.cc:1367] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.089s user 0.009s sys 0.021s
I20240507 08:11:50.348815 5654 leader_election.cc:302] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.349169 5851 raft_consensus.cc:2726] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:50.349404 5851 raft_consensus.cc:2726] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:50.349470 5843 raft_consensus.cc:348] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.349680 5843 raft_consensus.cc:374] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.349771 5843 raft_consensus.cc:729] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.349905 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 (DEFAULT_TABLE table=default.Table_8b96df6db03e4f77a97662d764097c4c [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.350268 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1cf2df0d3bd46268e22713fcc135a74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.350400 5837 tablet_replica.cc:330] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.350571 5837 raft_consensus.cc:2229] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.350728 5837 raft_consensus.cc:2258] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.352731 5843 consensus_queue.cc:260] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.353751 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d58aa20be2d34d0e9ec80656b522d017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.353919 5700 tablet_service.cc:1467] Processing CreateTablet for tablet fa932c6ad48f4ac183aabba8699cb148 (DEFAULT_TABLE table=default.Table_e3e2810edd06425882e48b9aeaada4f6 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.354301 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa932c6ad48f4ac183aabba8699cb148. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.354262 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d58aa20be2d34d0e9ec80656b522d017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.355177 5843 ts_tablet_manager.cc:1398] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:50.356305 5843 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.358062 5843 tablet_bootstrap.cc:654] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.295684 5840 consensus_queue.cc:260] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.358512 5840 ts_tablet_manager.cc:1398] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.089s user 0.000s sys 0.002s
I20240507 08:11:50.358814 5840 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.359982 5837 ts_tablet_manager.cc:1875] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.360337 5840 tablet_bootstrap.cc:654] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.363375 5695 tablet_service.cc:1467] Processing CreateTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 (DEFAULT_TABLE table=default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.363916 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d (DEFAULT_TABLE table=default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.366212 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f968f122c39b4fd5b995e8a2848a52d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.368818 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09adb78ad08a4e759eb9344cbabb6a9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.379407 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.379772 5841 tablet_replica.cc:330] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.380025 5841 raft_consensus.cc:2229] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.380218 5841 raft_consensus.cc:2258] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.381289 5841 ts_tablet_manager.cc:1875] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.383184 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 19efea8905cf4e83a523c278b2f16dc7
I20240507 08:11:50.383854 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 19efea8905cf4e83a523c278b2f16dc7 deleted and replaced by tablet add94bcf0c844cb78059b7240c7e28c0
I20240507 08:11:50.384407 5838 ts_tablet_manager.cc:1875] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.385677 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.387591 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 44f6f62c56cb4c29bf05914ca461a559
I20240507 08:11:50.388447 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.393358 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.393628 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.394153 5839 tablet_bootstrap.cc:492] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.394333 5839 ts_tablet_manager.cc:1367] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.010s
I20240507 08:11:50.395182 5839 raft_consensus.cc:348] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.395417 5839 raft_consensus.cc:374] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.395496 5839 raft_consensus.cc:729] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.395239 5841 ts_tablet_manager.cc:1888] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.395771 5839 consensus_queue.cc:260] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.396042 5841 log.cc:1196] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/a485c1dc1cae40fcadce8ce75a5607b2
I20240507 08:11:50.396513 5841 ts_tablet_manager.cc:1909] T a485c1dc1cae40fcadce8ce75a5607b2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.397729 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet a485c1dc1cae40fcadce8ce75a5607b2 (table default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]) successfully deleted
I20240507 08:11:50.397913 5839 ts_tablet_manager.cc:1398] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240507 08:11:50.398522 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 19efea8905cf4e83a523c278b2f16dc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19efea8905cf4e83a523c278b2f16dc7 already in progress: creating tablet
I20240507 08:11:50.399030 5839 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:50.399116 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 44f6f62c56cb4c29bf05914ca461a559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44f6f62c56cb4c29bf05914ca461a559 already in progress: creating tablet
I20240507 08:11:50.400658 5839 tablet_bootstrap.cc:654] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:50.401895 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 53144.5us
I20240507 08:11:50.402689 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.403082 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.401240 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ac1b4843bfc4cb491092f9b3204d8a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c68b6c08f00d46779a31f4e9eed99f9f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.403460 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.403589 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 9922d86933cc40dda1efff9ef03dcdf2 (table default.Table_b5e92e3768db4d958956b9730db5e8a8 [id=250c3443457344a2b78b762a432e28fe]) successfully deleted
I20240507 08:11:50.405604 5838 ts_tablet_manager.cc:1888] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.405777 5838 log.cc:1196] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:50.406483 5838 ts_tablet_manager.cc:1909] T d6d9ab293113410c857b5de903e21eb5 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.407446 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d6d9ab293113410c857b5de903e21eb5 (table default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]) successfully deleted
W20240507 08:11:50.409250 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 19efea8905cf4e83a523c278b2f16dc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19efea8905cf4e83a523c278b2f16dc7 already in progress: creating tablet
I20240507 08:11:50.410243 5840 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.411263 5840 ts_tablet_manager.cc:1367] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.011s
I20240507 08:11:50.412025 5840 raft_consensus.cc:348] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.412341 5840 raft_consensus.cc:374] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.412446 5840 raft_consensus.cc:729] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.412698 5840 consensus_queue.cc:260] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.413273 5840 ts_tablet_manager.cc:1398] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:11:50.414831 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 87e1f43426b14f98b985656f8d0d7141
I20240507 08:11:50.416694 5840 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.418220 5840 tablet_bootstrap.cc:654] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.419886 5838 tablet_replica.cc:330] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.419981 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.420214 5838 raft_consensus.cc:2229] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.420533 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:50.420670 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 44f6f62c56cb4c29bf05914ca461a559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44f6f62c56cb4c29bf05914ca461a559 already in progress: creating tablet
I20240507 08:11:50.420392 5838 raft_consensus.cc:2258] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.421738 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.422438 5838 ts_tablet_manager.cc:1875] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.428508 5837 ts_tablet_manager.cc:1888] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.428552 5843 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.428566 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8b96df6db03e4f77a97662d764097c4c" } new_table_name: "default.Table_38da0b3a42d940728e07debdd163315b" modify_external_catalogs: true
I20240507 08:11:50.428725 5843 ts_tablet_manager.cc:1367] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.073s user 0.019s sys 0.029s
I20240507 08:11:50.429132 5837 log.cc:1196] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e4a66fa7446a4dbea3885f8778e7f92b
I20240507 08:11:50.429664 5837 ts_tablet_manager.cc:1909] T e4a66fa7446a4dbea3885f8778e7f92b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.429534 5843 raft_consensus.cc:348] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.429805 5843 raft_consensus.cc:374] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.429908 5843 raft_consensus.cc:729] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:11:50.429908 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 87e1f43426b14f98b985656f8d0d7141 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e1f43426b14f98b985656f8d0d7141 already in progress: creating tablet
I20240507 08:11:50.429960 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dec664d2f47d45eb924462d597822a47" } new_table_name: "default.Table_65d6fd82bd8f4528ad808fb91dd27790" modify_external_catalogs: true
I20240507 08:11:50.430202 5843 consensus_queue.cc:260] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.433990 5843 ts_tablet_manager.cc:1398] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240507 08:11:50.434530 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 708d4bee30cc45bd9b86c56beb2c3ece from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.434659 5837 tablet_replica.cc:330] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.434697 5840 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.434859 5840 ts_tablet_manager.cc:1367] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.002s
I20240507 08:11:50.435217 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_678c1601578c46188487c197115995aa" } new_table_name: "default.Table_1b3804a3ddb546bdb6a9d9813e2372d0" modify_external_catalogs: true
I20240507 08:11:50.435570 5840 raft_consensus.cc:348] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.435741 5840 raft_consensus.cc:374] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.435842 5840 raft_consensus.cc:729] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.436080 5838 ts_tablet_manager.cc:1888] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.436051 5840 consensus_queue.cc:260] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.436204 5838 log.cc:1196] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1ac1b4843bfc4cb491092f9b3204d8a4
I20240507 08:11:50.436571 5838 ts_tablet_manager.cc:1909] T 1ac1b4843bfc4cb491092f9b3204d8a4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.434862 5837 raft_consensus.cc:2229] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.436882 5837 raft_consensus.cc:2258] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.437116 5843 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.437507 5837 ts_tablet_manager.cc:1875] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.437608 5840 ts_tablet_manager.cc:1398] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240507 08:11:50.438635 5843 tablet_bootstrap.cc:654] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.438661 5840 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.440183 5840 tablet_bootstrap.cc:654] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.440245 5839 tablet_bootstrap.cc:492] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.440397 5839 ts_tablet_manager.cc:1367] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.011s
I20240507 08:11:50.441226 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:50.441201 5839 raft_consensus.cc:348] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.441399 5839 raft_consensus.cc:374] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.441510 5839 raft_consensus.cc:729] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:11:50.441762 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 87e1f43426b14f98b985656f8d0d7141 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e1f43426b14f98b985656f8d0d7141 already in progress: creating tablet
I20240507 08:11:50.441768 5839 consensus_queue.cc:260] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.442837 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e4a66fa7446a4dbea3885f8778e7f92b (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
W20240507 08:11:50.443241 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5cb62017acbd4472ac5b143eafb36ae9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cb62017acbd4472ac5b143eafb36ae9 already in progress: creating tablet
I20240507 08:11:50.443375 5839 ts_tablet_manager.cc:1398] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:11:50.444450 5839 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:50.444638 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 717460af19164c058f42c7a03095bb30 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 717460af19164c058f42c7a03095bb30 already in progress: creating tablet
I20240507 08:11:50.446761 5839 tablet_bootstrap.cc:654] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.446748 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8aaa2c0a5ef44d2bb4718bbc70104607" } modify_external_catalogs: true
I20240507 08:11:50.446965 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8aaa2c0a5ef44d2bb4718bbc70104607" } modify_external_catalogs: true
I20240507 08:11:50.447232 5837 ts_tablet_manager.cc:1888] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.447396 5837 log.cc:1196] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/aa1942369ba1413cb1c1198d4c972234
I20240507 08:11:50.447728 5837 ts_tablet_manager.cc:1909] T aa1942369ba1413cb1c1198d4c972234 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.448364 5839 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.448523 5839 ts_tablet_manager.cc:1367] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
W20240507 08:11:50.449764 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1cf2df0d3bd46268e22713fcc135a74 consensus state has no leader
I20240507 08:11:50.450568 5837 tablet_replica.cc:330] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.450618 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7 [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 1 got 0
I20240507 08:11:50.450737 5837 raft_consensus.cc:2229] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.450911 5837 raft_consensus.cc:2258] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.450898 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_dec664d2f47d45eb924462d597822a47 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 3 got 0
I20240507 08:11:50.451604 5837 ts_tablet_manager.cc:1875] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.451917 5839 raft_consensus.cc:348] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.452155 5839 raft_consensus.cc:374] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.452227 5839 raft_consensus.cc:729] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.453169 5843 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.453313 5843 ts_tablet_manager.cc:1367] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240507 08:11:50.454165 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 6616660bc4a6400a9eb644590ced0e21 (DEFAULT_TABLE table=default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e [id=e5e2009b88a343018ab45566f5d2246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.454144 5843 raft_consensus.cc:348] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.454344 5843 raft_consensus.cc:374] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.454291 5262 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_dec664d2f47d45eb924462d597822a47 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 3 got 0
I20240507 08:11:50.454428 5843 raft_consensus.cc:729] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.454573 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616660bc4a6400a9eb644590ced0e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.454679 5843 consensus_queue.cc:260] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:50.457234 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 91c6cb7920df47aa97be6555c587798a (table default.Table_25edacce85f54417862546993cc8d161 [id=feeaf98e940f4006814716f3df172e51], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 91c6cb7920df47aa97be6555c587798a consensus state has no leader
I20240507 08:11:50.457587 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet aa1942369ba1413cb1c1198d4c972234 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
W20240507 08:11:50.457829 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d6d9ab293113410c857b5de903e21eb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6d9ab293113410c857b5de903e21eb5 already in progress: creating tablet
W20240507 08:11:50.458047 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 87ede5b03251463d8608d15f037e4484 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87ede5b03251463d8608d15f037e4484 already in progress: creating tablet
I20240507 08:11:50.458122 5843 ts_tablet_manager.cc:1398] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:11:50.458139 5837 ts_tablet_manager.cc:1888] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:11:50.458251 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c68b6c08f00d46779a31f4e9eed99f9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c68b6c08f00d46779a31f4e9eed99f9f already in progress: creating tablet
I20240507 08:11:50.458282 5837 log.cc:1196] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/86b8cfe64e5f418bbfb65087003beb66
W20240507 08:11:50.458537 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c9b8044e4a0a41218f67513b14c0ebeb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b8044e4a0a41218f67513b14c0ebeb already in progress: creating tablet
I20240507 08:11:50.458760 5837 ts_tablet_manager.cc:1909] T 86b8cfe64e5f418bbfb65087003beb66 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.459331 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7 [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 1 got 0
I20240507 08:11:50.459908 5837 tablet_replica.cc:330] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.460101 5837 raft_consensus.cc:2229] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.460314 5843 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.460356 5837 raft_consensus.cc:2258] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.460572 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 86b8cfe64e5f418bbfb65087003beb66 (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
W20240507 08:11:50.460829 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ed0074cc70e248ac8b5e90cf57b441c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0074cc70e248ac8b5e90cf57b441c3 already in progress: creating tablet
W20240507 08:11:50.462467 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34ffe98d424a4eb5a08be067f96a50f9 consensus state has no leader
I20240507 08:11:50.462895 5843 tablet_bootstrap.cc:654] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.463841 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0514ff21565c4cc394b734007163903e" } new_table_name: "default.Table_8e33d7020c354b828f678dbbd57d5832" modify_external_catalogs: true
I20240507 08:11:50.464236 5837 ts_tablet_manager.cc:1875] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.464612 5839 consensus_queue.cc:260] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:50.465021 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 has no consensus state
I20240507 08:11:50.465873 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 6616660bc4a6400a9eb644590ced0e21 (DEFAULT_TABLE table=default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e [id=e5e2009b88a343018ab45566f5d2246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.466231 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616660bc4a6400a9eb644590ced0e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.466177 5419 tablet_service.cc:1467] Processing CreateTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.466533 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f3ee2b04de4cdf8a621b39f1476577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.467226 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 708d4bee30cc45bd9b86c56beb2c3ece
I20240507 08:11:50.467634 5261 catalog_manager.cc:6554] ReplaceTablet: tablet 708d4bee30cc45bd9b86c56beb2c3ece deleted and replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3
I20240507 08:11:50.468056 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1" } modify_external_catalogs: true
I20240507 08:11:50.468220 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1" } modify_external_catalogs: true
I20240507 08:11:50.468845 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 6616660bc4a6400a9eb644590ced0e21 (DEFAULT_TABLE table=default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e [id=e5e2009b88a343018ab45566f5d2246c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.469183 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6616660bc4a6400a9eb644590ced0e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.469983 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:50.470685 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 708d4bee30cc45bd9b86c56beb2c3ece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 708d4bee30cc45bd9b86c56beb2c3ece already in progress: creating tablet
I20240507 08:11:50.471654 5839 ts_tablet_manager.cc:1398] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240507 08:11:50.472894 5839 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.474586 5839 tablet_bootstrap.cc:654] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.474975 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 (DEFAULT_TABLE table=default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.475827 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.476217 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 (DEFAULT_TABLE table=default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.476586 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a35991c18414b9bc714e9a0ba14d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.476972 5562 tablet_service.cc:1467] Processing CreateTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.477784 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.477906 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 91c6cb7920df47aa97be6555c587798a (DEFAULT_TABLE table=default.Table_25edacce85f54417862546993cc8d161 [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.478447 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f3ee2b04de4cdf8a621b39f1476577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.479260 5699 tablet_service.cc:1467] Processing CreateTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.479676 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0f3ee2b04de4cdf8a621b39f1476577. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.483208 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.483418 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f968f122c39b4fd5b995e8a2848a52d4
I20240507 08:11:50.483561 5841 tablet_replica.cc:330] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.483724 5841 raft_consensus.cc:2229] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.483942 5841 raft_consensus.cc:2258] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:50.484494 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 708d4bee30cc45bd9b86c56beb2c3ece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 708d4bee30cc45bd9b86c56beb2c3ece already in progress: creating tablet
I20240507 08:11:50.485047 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.485632 5260 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_65d6fd82bd8f4528ad808fb91dd27790 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 4 got 0
I20240507 08:11:50.485709 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 91c6cb7920df47aa97be6555c587798a (DEFAULT_TABLE table=default.Table_25edacce85f54417862546993cc8d161 [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.486093 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c6cb7920df47aa97be6555c587798a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.489302 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a5f462941b57438eb01a3cc87a0390bc" } new_table_name: "default.Table_6348e5422e1d4fedaac4c6084f846b96" modify_external_catalogs: true
I20240507 08:11:50.489843 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 (DEFAULT_TABLE table=default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.490234 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a35991c18414b9bc714e9a0ba14d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.490259 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_65d6fd82bd8f4528ad808fb91dd27790 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 4 got 0
I20240507 08:11:50.492686 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 91c6cb7920df47aa97be6555c587798a (DEFAULT_TABLE table=default.Table_25edacce85f54417862546993cc8d161 [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.493109 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c6cb7920df47aa97be6555c587798a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.493589 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 400d2a7b0e224e46a5a580ca22e96eab consensus state has no leader
I20240507 08:11:50.494443 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.494699 5838 tablet_replica.cc:330] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.494730 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b650283bfd9b448c8a0ba81f514f2831" } modify_external_catalogs: true
I20240507 08:11:50.494876 5838 raft_consensus.cc:2229] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.494931 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b650283bfd9b448c8a0ba81f514f2831" } modify_external_catalogs: true
I20240507 08:11:50.495054 5838 raft_consensus.cc:2258] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.495440 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6a35991c18414b9bc714e9a0ba14d8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.496147 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f968f122c39b4fd5b995e8a2848a52d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f968f122c39b4fd5b995e8a2848a52d4 already in progress: creating tablet
I20240507 08:11:50.496765 5838 ts_tablet_manager.cc:1875] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.496656 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f053f92b620148968b86e702410b211d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.497186 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f053f92b620148968b86e702410b211d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.498970 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4c3d879d3da3408ea46e6ae4ab672eca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.499567 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c3d879d3da3408ea46e6ae4ab672eca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.479698 5837 ts_tablet_manager.cc:1888] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.501992 5839 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.502192 5839 ts_tablet_manager.cc:1367] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.015s
I20240507 08:11:50.503569 5839 raft_consensus.cc:348] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.504456 5839 raft_consensus.cc:374] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.504583 5839 raft_consensus.cc:729] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.504753 5841 ts_tablet_manager.cc:1875] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.505159 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.505923 5838 ts_tablet_manager.cc:1888] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.506060 5838 log.cc:1196] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:50.505163 5839 consensus_queue.cc:260] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.506418 5838 ts_tablet_manager.cc:1909] T 71bbe28a013344bcb6529247d7139ab9 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.508078 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 71bbe28a013344bcb6529247d7139ab9 (table default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]) successfully deleted
I20240507 08:11:50.509608 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:50.509721 5839 ts_tablet_manager.cc:1398] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240507 08:11:50.511966 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.513897 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c6cb7920df47aa97be6555c587798a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.514624 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.516793 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 717460af19164c058f42c7a03095bb30 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.514624 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.519251 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.519511 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.522480 5838 tablet_replica.cc:330] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.522766 5838 raft_consensus.cc:2229] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.522897 5846 raft_consensus.cc:482] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.523027 5838 raft_consensus.cc:2258] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.523064 5846 raft_consensus.cc:504] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.523725 5846 leader_election.cc:288] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.523838 5838 ts_tablet_manager.cc:1875] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.524451 5841 ts_tablet_manager.cc:1888] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.524577 5841 log.cc:1196] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5cb62017acbd4472ac5b143eafb36ae9
I20240507 08:11:50.524945 5841 ts_tablet_manager.cc:1909] T 5cb62017acbd4472ac5b143eafb36ae9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.525111 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.525381 5585 raft_consensus.cc:2454] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:50.525933 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5cb62017acbd4472ac5b143eafb36ae9 (table default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]) successfully deleted
I20240507 08:11:50.525931 5378 leader_election.cc:302] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:11:50.526243 5846 raft_consensus.cc:2781] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.526345 5846 raft_consensus.cc:482] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.526438 5846 raft_consensus.cc:3037] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.526654 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:50.526932 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f968f122c39b4fd5b995e8a2848a52d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f968f122c39b4fd5b995e8a2848a52d4 already in progress: creating tablet
W20240507 08:11:50.527467 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0a6a35991c18414b9bc714e9a0ba14d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a35991c18414b9bc714e9a0ba14d8 already in progress: creating tablet
W20240507 08:11:50.527863 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d10b38120dfe4b9ab3c39cf584e4ff13 has no consensus state
I20240507 08:11:50.528682 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 5cfe3c25a9a84eab9091502955453951 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.529148 5846 raft_consensus.cc:504] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.529656 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.529896 5719 raft_consensus.cc:2454] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:50.531195 5839 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.532850 5839 tablet_bootstrap.cc:654] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.533021 5837 log.cc:1196] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1ac1b4843bfc4cb491092f9b3204d8a4
I20240507 08:11:50.533474 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8ea5220905954dc9b078e00e0db69fd2
I20240507 08:11:50.533526 5837 ts_tablet_manager.cc:1909] T 1ac1b4843bfc4cb491092f9b3204d8a4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.534338 5846 leader_election.cc:288] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.534637 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:50.534859 5719 raft_consensus.cc:3037] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.535095 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1ac1b4843bfc4cb491092f9b3204d8a4 (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:50.535899 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:50.536435 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 19efea8905cf4e83a523c278b2f16dc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19efea8905cf4e83a523c278b2f16dc7 already in progress: creating tablet
W20240507 08:11:50.537039 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 44f6f62c56cb4c29bf05914ca461a559 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44f6f62c56cb4c29bf05914ca461a559 already in progress: creating tablet
W20240507 08:11:50.537621 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 87e1f43426b14f98b985656f8d0d7141 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87e1f43426b14f98b985656f8d0d7141 already in progress: creating tablet
I20240507 08:11:50.537876 5719 raft_consensus.cc:2454] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:50.536873 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.538252 5377 leader_election.cc:302] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:50.538569 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 708d4bee30cc45bd9b86c56beb2c3ece because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 708d4bee30cc45bd9b86c56beb2c3ece already in progress: creating tablet
I20240507 08:11:50.533875 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:50.534485 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.538987 5901 raft_consensus.cc:2781] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.539041 5585 raft_consensus.cc:3037] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:50.539486 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8ea5220905954dc9b078e00e0db69fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea5220905954dc9b078e00e0db69fd2 already in progress: creating tablet
W20240507 08:11:50.539551 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0a6a35991c18414b9bc714e9a0ba14d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a35991c18414b9bc714e9a0ba14d8 already in progress: creating tablet
W20240507 08:11:50.540606 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f968f122c39b4fd5b995e8a2848a52d4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f968f122c39b4fd5b995e8a2848a52d4 already in progress: creating tablet
I20240507 08:11:50.541095 5837 tablet_replica.cc:330] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:50.541254 5837 raft_consensus.cc:2229] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.541410 5837 raft_consensus.cc:2258] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.541980 5585 raft_consensus.cc:2454] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:50.543121 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a10d53fdc3bc4f009010ffe48156725d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.543694 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a10d53fdc3bc4f009010ffe48156725d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.543946 5837 ts_tablet_manager.cc:1875] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.548224 5843 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.548513 5843 ts_tablet_manager.cc:1367] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.088s user 0.010s sys 0.015s
I20240507 08:11:50.549340 5843 raft_consensus.cc:348] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.549661 5843 raft_consensus.cc:374] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.549777 5843 raft_consensus.cc:729] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.550657 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 (DEFAULT_TABLE table=default.Table_d58aa20be2d34d0e9ec80656b522d017 [id=a764e2dd5a9841fe9ebc72c179b7858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.551069 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812dd1ebd42f4e929838a0535c22ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.551807 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 (DEFAULT_TABLE table=default.Table_d58aa20be2d34d0e9ec80656b522d017 [id=a764e2dd5a9841fe9ebc72c179b7858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.551882 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 (DEFAULT_TABLE table=default.Table_d58aa20be2d34d0e9ec80656b522d017 [id=a764e2dd5a9841fe9ebc72c179b7858e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.552186 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812dd1ebd42f4e929838a0535c22ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.552253 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (DEFAULT_TABLE table=default.Table_cdb3e713b1194129bd3f5be0ac3991e6 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.552294 5843 consensus_queue.cc:260] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.552258 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 812dd1ebd42f4e929838a0535c22ad69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.554664 5843 ts_tablet_manager.cc:1398] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:11:50.555920 5843 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.557614 5843 tablet_bootstrap.cc:654] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.557716 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.557898 5565 tablet_service.cc:1467] Processing CreateTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 (DEFAULT_TABLE table=default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.558691 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10b38120dfe4b9ab3c39cf584e4ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.559523 5420 tablet_service.cc:1467] Processing CreateTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 (DEFAULT_TABLE table=default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.560050 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10b38120dfe4b9ab3c39cf584e4ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.560830 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6c5e09f35af9471b89b6f760b282b361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.561373 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c5e09f35af9471b89b6f760b282b361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.562551 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (DEFAULT_TABLE table=default.Table_cdb3e713b1194129bd3f5be0ac3991e6 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.562737 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (DEFAULT_TABLE table=default.Table_cdb3e713b1194129bd3f5be0ac3991e6 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.562902 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.563117 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.568857 5695 tablet_service.cc:1467] Processing CreateTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 (DEFAULT_TABLE table=default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.569700 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d10b38120dfe4b9ab3c39cf584e4ff13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.570026 5843 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.570175 5843 ts_tablet_manager.cc:1367] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
I20240507 08:11:50.570958 5843 raft_consensus.cc:348] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.573241 5843 raft_consensus.cc:374] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.573655 5843 raft_consensus.cc:729] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.574085 5843 consensus_queue.cc:260] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.575032 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.575443 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.576275 5843 ts_tablet_manager.cc:1398] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:11:50.578794 5843 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.578984 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_40c0cbe3de2c4a28b7ac07e4245bb230"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.579531 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40c0cbe3de2c4a28b7ac07e4245bb230 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.580312 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.580463 5843 tablet_bootstrap.cc:654] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.580713 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.587317 5843 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.587495 5843 ts_tablet_manager.cc:1367] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240507 08:11:50.588670 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.589648 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.591564 5700 tablet_service.cc:1467] Processing CreateTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 (DEFAULT_TABLE table=default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.592311 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc6cc72bb7e4721a8ab559102273db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.594192 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5a63960776644f8186340cc40315f99f" } modify_external_catalogs: true
I20240507 08:11:50.594374 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 2cfa9a247eba402cb3f0b818ad9f837a from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.594473 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5a63960776644f8186340cc40315f99f" } modify_external_catalogs: true
I20240507 08:11:50.590276 5843 raft_consensus.cc:348] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.594663 5843 raft_consensus.cc:374] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.594764 5843 raft_consensus.cc:729] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.595062 5843 consensus_queue.cc:260] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.596931 5843 ts_tablet_manager.cc:1398] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:11:50.596951 5840 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.597110 5840 ts_tablet_manager.cc:1367] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.159s user 0.001s sys 0.005s
I20240507 08:11:50.598104 5843 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.598379 5840 raft_consensus.cc:348] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.598695 5840 raft_consensus.cc:374] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.598825 5840 raft_consensus.cc:729] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.600065 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2cfa9a247eba402cb3f0b818ad9f837a
I20240507 08:11:50.600555 5843 tablet_bootstrap.cc:654] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.600708 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 2cfa9a247eba402cb3f0b818ad9f837a deleted and replaced by tablet 697611671ef64c53b0be4f5341b8f270
I20240507 08:11:50.600764 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.594450 5421 tablet_service.cc:1467] Processing CreateTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 (DEFAULT_TABLE table=default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.601107 5840 consensus_queue.cc:260] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.601275 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.602784 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc6cc72bb7e4721a8ab559102273db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.603929 5891 raft_consensus.cc:482] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.604097 5891 raft_consensus.cc:504] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.604663 5562 tablet_service.cc:1467] Processing CreateTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 (DEFAULT_TABLE table=default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.605031 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebc6cc72bb7e4721a8ab559102273db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.605013 5891 leader_election.cc:288] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:50.605723 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:50.605974 5447 raft_consensus.cc:2454] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:50.606467 5656 leader_election.cc:302] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:50.607784 5891 raft_consensus.cc:2781] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.607801 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 5405f264bac1464a8f1b4435715a64a4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.607920 5891 raft_consensus.cc:482] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.608032 5891 raft_consensus.cc:3037] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.608168 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405f264bac1464a8f1b4435715a64a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.609313 5424 tablet_service.cc:1467] Processing CreateTablet for tablet add94bcf0c844cb78059b7240c7e28c0 (DEFAULT_TABLE table=default.Table_a19ae1cf5dd64ba0bd8424ab78f287e4 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.609691 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add94bcf0c844cb78059b7240c7e28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.610934 5891 raft_consensus.cc:504] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.611975 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:50.612179 5447 raft_consensus.cc:3037] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.613901 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 5cab176c4bd34f38b14ac096f79b912e (DEFAULT_TABLE table=default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.614006 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 (DEFAULT_TABLE table=default.Table_745f979f496a402fb4868ac8ed7099ff [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.614225 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cab176c4bd34f38b14ac096f79b912e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.614428 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228590c8e7d4c34b3acab9eb16665d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.614579 5837 ts_tablet_manager.cc:1888] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.614688 5837 log.cc:1196] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5cfe3c25a9a84eab9091502955453951
I20240507 08:11:50.615017 5447 raft_consensus.cc:2454] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:50.615062 5837 ts_tablet_manager.cc:1909] T 5cfe3c25a9a84eab9091502955453951 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:50.615732 5656 leader_election.cc:302] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:50.616556 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c4899c088531480ea15364af94595ee5 (DEFAULT_TABLE table=default.Table_4e6dbc6b805d4d6888de30dc35f6972b [id=307b8b3612c1470d916ad0f37c986171]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.616981 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4899c088531480ea15364af94595ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.617192 5851 raft_consensus.cc:2781] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.617408 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:50.618467 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 (DEFAULT_TABLE table=default.Table_745f979f496a402fb4868ac8ed7099ff [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.618851 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228590c8e7d4c34b3acab9eb16665d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.619671 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.621035 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 5405f264bac1464a8f1b4435715a64a4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.621387 5654 leader_election.cc:341] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.621474 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405f264bac1464a8f1b4435715a64a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.621742 5694 tablet_service.cc:1467] Processing CreateTablet for tablet add94bcf0c844cb78059b7240c7e28c0 (DEFAULT_TABLE table=default.Table_a19ae1cf5dd64ba0bd8424ab78f287e4 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.621814 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 5cab176c4bd34f38b14ac096f79b912e (DEFAULT_TABLE table=default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.622097 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add94bcf0c844cb78059b7240c7e28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.622552 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cab176c4bd34f38b14ac096f79b912e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.622735 5891 leader_election.cc:288] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:50.625594 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:50.625855 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 (DEFAULT_TABLE table=default.Table_745f979f496a402fb4868ac8ed7099ff [id=2b718529a9e7440d831c3d60a8c17ba4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.626201 5654 leader_election.cc:341] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.626876 5697 tablet_service.cc:1467] Processing CreateTablet for tablet c4899c088531480ea15364af94595ee5 (DEFAULT_TABLE table=default.Table_4e6dbc6b805d4d6888de30dc35f6972b [id=307b8b3612c1470d916ad0f37c986171]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.627724 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:50.626209 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5228590c8e7d4c34b3acab9eb16665d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.628710 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4899c088531480ea15364af94595ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.632509 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 5405f264bac1464a8f1b4435715a64a4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.632854 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5405f264bac1464a8f1b4435715a64a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.633975 5564 tablet_service.cc:1467] Processing CreateTablet for tablet add94bcf0c844cb78059b7240c7e28c0 (DEFAULT_TABLE table=default.Table_a19ae1cf5dd64ba0bd8424ab78f287e4 [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.634313 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet add94bcf0c844cb78059b7240c7e28c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.634510 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_431fe83bbf04464e84771e65abee1c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.635301 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_431fe83bbf04464e84771e65abee1c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.637115 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.637893 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.637869 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b7024e203f50440095d2cf53ca83c775" } modify_external_catalogs: true
I20240507 08:11:50.638082 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b7024e203f50440095d2cf53ca83c775" } modify_external_catalogs: true
I20240507 08:11:50.638389 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5cfe3c25a9a84eab9091502955453951 (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
W20240507 08:11:50.638876 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8ea5220905954dc9b078e00e0db69fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea5220905954dc9b078e00e0db69fd2 already in progress: creating tablet
W20240507 08:11:50.639113 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2cfa9a247eba402cb3f0b818ad9f837a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cfa9a247eba402cb3f0b818ad9f837a already in progress: creating tablet
I20240507 08:11:50.639835 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.639868 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2f5246f5a6874c848d060c15dd9be10f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:50.639957 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 5cab176c4bd34f38b14ac096f79b912e (DEFAULT_TABLE table=default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.640290 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5cab176c4bd34f38b14ac096f79b912e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.640341 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f5246f5a6874c848d060c15dd9be10f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.640643 5840 ts_tablet_manager.cc:1398] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20240507 08:11:50.641364 5560 tablet_service.cc:1467] Processing CreateTablet for tablet c4899c088531480ea15364af94595ee5 (DEFAULT_TABLE table=default.Table_4e6dbc6b805d4d6888de30dc35f6972b [id=307b8b3612c1470d916ad0f37c986171]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:50.641796 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 already in progress: creating tablet
I20240507 08:11:50.641885 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4899c088531480ea15364af94595ee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.642290 5840 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.644308 5840 tablet_bootstrap.cc:654] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.645964 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.646759 5901 raft_consensus.cc:686] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
W20240507 08:11:50.647145 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2cfa9a247eba402cb3f0b818ad9f837a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cfa9a247eba402cb3f0b818ad9f837a already in progress: creating tablet
I20240507 08:11:50.647285 5901 consensus_queue.cc:237] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.649664 5838 ts_tablet_manager.cc:1888] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.649842 5838 log.cc:1196] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/717460af19164c058f42c7a03095bb30
I20240507 08:11:50.650264 5838 ts_tablet_manager.cc:1909] T 717460af19164c058f42c7a03095bb30 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.652166 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_faabc2fd2afc43b391324d3e7898c00c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.652468 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 already in progress: creating tablet
W20240507 08:11:50.652705 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faabc2fd2afc43b391324d3e7898c00c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.653401 5839 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.653592 5839 ts_tablet_manager.cc:1367] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.122s user 0.013s sys 0.027s
I20240507 08:11:50.653883 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4ea4bbd2ff0a49a89be1371511b5986c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.654330 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea4bbd2ff0a49a89be1371511b5986c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.654379 5839 raft_consensus.cc:348] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.654642 5839 raft_consensus.cc:374] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.654713 5839 raft_consensus.cc:729] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.655055 5839 consensus_queue.cc:260] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.655391 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e97ccca6ca324143a7e713d916068114"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.655817 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e97ccca6ca324143a7e713d916068114 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.656800 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_25edacce85f54417862546993cc8d161" } new_table_name: "default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c" modify_external_catalogs: true
I20240507 08:11:50.658691 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7 [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 1 got 0
I20240507 08:11:50.660750 5839 ts_tablet_manager.cc:1398] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240507 08:11:50.662498 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7 [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 1 got 0
I20240507 08:11:50.665831 5839 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.667984 5839 tablet_bootstrap.cc:654] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.669631 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a19ae1cf5dd64ba0bd8424ab78f287e4" } new_table_name: "default.Table_237e57be541d40359403f08fcc681dd8" modify_external_catalogs: true
I20240507 08:11:50.669960 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1666cbd5c04f43f4aca54ecbed94ec7a" } new_table_name: "default.Table_11f043808d86473e9d1ff0c532d9c4c0" modify_external_catalogs: true
I20240507 08:11:50.672751 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 717460af19164c058f42c7a03095bb30 (table default.Table_8051479eeaf74dd8beccf13762a64249 [id=5d190d55ad4748cca262a1a33e12b68a]) successfully deleted
W20240507 08:11:50.673257 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0a6a35991c18414b9bc714e9a0ba14d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0a6a35991c18414b9bc714e9a0ba14d8 already in progress: creating tablet
I20240507 08:11:50.676784 5843 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.676977 5843 ts_tablet_manager.cc:1367] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.079s user 0.008s sys 0.012s
I20240507 08:11:50.677923 5840 tablet_bootstrap.cc:492] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.677860 5843 raft_consensus.cc:348] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.678052 5843 raft_consensus.cc:374] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.678092 5840 ts_tablet_manager.cc:1367] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.015s
I20240507 08:11:50.678159 5843 raft_consensus.cc:729] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.678387 5843 consensus_queue.cc:260] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.679014 5843 ts_tablet_manager.cc:1398] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:50.678884 5840 raft_consensus.cc:348] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.679105 5840 raft_consensus.cc:374] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.679208 5840 raft_consensus.cc:729] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.679466 5840 consensus_queue.cc:260] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.680979 5840 ts_tablet_manager.cc:1398] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:50.682240 5840 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:50.683498 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8ea5220905954dc9b078e00e0db69fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ea5220905954dc9b078e00e0db69fd2 already in progress: creating tablet
W20240507 08:11:50.683717 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2cfa9a247eba402cb3f0b818ad9f837a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cfa9a247eba402cb3f0b818ad9f837a already in progress: creating tablet
W20240507 08:11:50.683936 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 already in progress: creating tablet
I20240507 08:11:50.684087 5840 tablet_bootstrap.cc:654] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.684886 5843 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.686966 5843 tablet_bootstrap.cc:654] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.688956 5843 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.689036 5839 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.689214 5843 ts_tablet_manager.cc:1367] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240507 08:11:50.689239 5839 ts_tablet_manager.cc:1367] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20240507 08:11:50.690061 5843 raft_consensus.cc:348] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.690332 5843 raft_consensus.cc:374] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.690439 5843 raft_consensus.cc:729] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.690308 5839 raft_consensus.cc:348] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.690591 5839 raft_consensus.cc:374] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.690711 5839 raft_consensus.cc:729] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.690851 5843 consensus_queue.cc:260] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.691576 5843 ts_tablet_manager.cc:1398] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:11:50.692302 5840 tablet_bootstrap.cc:492] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.692464 5840 ts_tablet_manager.cc:1367] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240507 08:11:50.692801 5843 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.693177 5840 raft_consensus.cc:348] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.693409 5840 raft_consensus.cc:374] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.693493 5840 raft_consensus.cc:729] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.693768 5840 consensus_queue.cc:260] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.694737 5843 tablet_bootstrap.cc:654] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.695528 5840 ts_tablet_manager.cc:1398] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:50.696141 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 87e1f43426b14f98b985656f8d0d7141 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.696599 5840 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.691023 5839 consensus_queue.cc:260] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.697973 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 34ffe98d424a4eb5a08be067f96a50f9 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.698076 5840 tablet_bootstrap.cc:654] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.700798 5839 ts_tablet_manager.cc:1398] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240507 08:11:50.701170 5840 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.701285 5840 ts_tablet_manager.cc:1367] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:11:50.702127 5840 raft_consensus.cc:348] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.702262 5901 raft_consensus.cc:482] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.702337 5840 raft_consensus.cc:374] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.702415 5840 raft_consensus.cc:729] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.702425 5901 raft_consensus.cc:504] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.702653 5840 consensus_queue.cc:260] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.703521 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81919c8f720f4c72bc2cc457b095859b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.703871 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81919c8f720f4c72bc2cc457b095859b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:50.704396 5377 leader_election.cc:341] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:50.704581 5378 leader_election.cc:341] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.704695 5378 leader_election.cc:302] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.704967 5902 raft_consensus.cc:2726] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:50.707837 5839 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.704581 5901 leader_election.cc:288] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.709830 5839 tablet_bootstrap.cc:654] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.710728 5840 ts_tablet_manager.cc:1398] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240507 08:11:50.711263 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1eb6ce65380844798cfe02364195022d" } modify_external_catalogs: true
I20240507 08:11:50.711529 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1eb6ce65380844798cfe02364195022d" } modify_external_catalogs: true
I20240507 08:11:50.712610 5840 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.713613 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 81919c8f720f4c72bc2cc457b095859b
I20240507 08:11:50.714587 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 81919c8f720f4c72bc2cc457b095859b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:50.715373 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 81919c8f720f4c72bc2cc457b095859b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81919c8f720f4c72bc2cc457b095859b already in progress: creating tablet
I20240507 08:11:50.715637 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7ebba49d572e49cfb4c8df58c1c1ea5e" } new_table_name: "default.Table_f485dedc2a6849ff8b388a9b07e06b73" modify_external_catalogs: true
I20240507 08:11:50.716817 5840 tablet_bootstrap.cc:654] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.717156 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 81919c8f720f4c72bc2cc457b095859b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:50.717404 5841 tablet_replica.cc:330] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:50.717617 5841 raft_consensus.cc:2229] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.717777 5841 raft_consensus.cc:2258] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.718076 5839 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.718523 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.720757 5839 ts_tablet_manager.cc:1367] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20240507 08:11:50.721050 5838 tablet_replica.cc:330] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.721235 5838 raft_consensus.cc:2229] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.721422 5838 raft_consensus.cc:2258] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.721972 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 81919c8f720f4c72bc2cc457b095859b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.723367 5841 ts_tablet_manager.cc:1875] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.722692 5838 ts_tablet_manager.cc:1875] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.727511 5840 tablet_bootstrap.cc:492] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.727699 5840 ts_tablet_manager.cc:1367] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240507 08:11:50.729683 5840 raft_consensus.cc:348] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.729949 5840 raft_consensus.cc:374] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.730041 5840 raft_consensus.cc:729] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.730731 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8e33d7020c354b828f678dbbd57d5832" } modify_external_catalogs: true
I20240507 08:11:50.730923 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8e33d7020c354b828f678dbbd57d5832" } modify_external_catalogs: true
I20240507 08:11:50.731663 5841 ts_tablet_manager.cc:1888] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.731587 5840 consensus_queue.cc:260] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.731823 5841 log.cc:1196] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/81919c8f720f4c72bc2cc457b095859b
I20240507 08:11:50.732229 5841 ts_tablet_manager.cc:1909] T 81919c8f720f4c72bc2cc457b095859b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:50.733294 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 81919c8f720f4c72bc2cc457b095859b (table default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]) successfully deleted
I20240507 08:11:50.733521 5840 ts_tablet_manager.cc:1398] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:11:50.733562 5843 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.733716 5843 ts_tablet_manager.cc:1367] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.004s
I20240507 08:11:50.734071 5838 ts_tablet_manager.cc:1888] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.734184 5838 log.cc:1196] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/a485c1dc1cae40fcadce8ce75a5607b2
I20240507 08:11:50.734452 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 0a6a35991c18414b9bc714e9a0ba14d8 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.734421 5843 raft_consensus.cc:348] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.734620 5843 raft_consensus.cc:374] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.734736 5843 raft_consensus.cc:729] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.735000 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c5617bc7e44f49f6aa3dfee0f6ff79b7" } new_table_name: "default.Table_cb5f53bbcf3049939314d98e891fe83c" modify_external_catalogs: true
I20240507 08:11:50.734998 5843 consensus_queue.cc:260] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.722357 5839 raft_consensus.cc:348] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.735234 5839 raft_consensus.cc:374] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.735318 5839 raft_consensus.cc:729] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.735497 5843 ts_tablet_manager.cc:1398] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:11:50.735589 5839 consensus_queue.cc:260] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.735790 5843 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.736042 5838 ts_tablet_manager.cc:1909] T a485c1dc1cae40fcadce8ce75a5607b2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:50.737350 5843 tablet_bootstrap.cc:654] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.738257 5840 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.741058 5840 tablet_bootstrap.cc:654] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.743431 5839 ts_tablet_manager.cc:1398] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240507 08:11:50.746815 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cebf0395fd3b4cf1a49263069e98ae91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.747387 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cebf0395fd3b4cf1a49263069e98ae91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.747680 5843 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.747830 5843 ts_tablet_manager.cc:1367] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240507 08:11:50.748533 5843 raft_consensus.cc:348] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.748740 5843 raft_consensus.cc:374] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.748819 5843 raft_consensus.cc:729] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.749082 5843 consensus_queue.cc:260] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.753371 5843 ts_tablet_manager.cc:1398] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
W20240507 08:11:50.755192 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91c6cb7920df47aa97be6555c587798a consensus state has no leader
I20240507 08:11:50.756371 5843 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.756661 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_faabc2fd2afc43b391324d3e7898c00c" } modify_external_catalogs: true
I20240507 08:11:50.756814 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_faabc2fd2afc43b391324d3e7898c00c" } modify_external_catalogs: true
I20240507 08:11:50.757973 5843 tablet_bootstrap.cc:654] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.759392 5839 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.759658 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet a485c1dc1cae40fcadce8ce75a5607b2 (table default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]) successfully deleted
W20240507 08:11:50.759923 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 81919c8f720f4c72bc2cc457b095859b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81919c8f720f4c72bc2cc457b095859b already in progress: creating tablet
I20240507 08:11:50.760665 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.761050 5838 tablet_replica.cc:330] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:50.761400 5838 raft_consensus.cc:2229] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:50.761801 5839 tablet_bootstrap.cc:654] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.761667 5838 raft_consensus.cc:2258] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:50.765077 5840 tablet_bootstrap.cc:492] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.765280 5840 ts_tablet_manager.cc:1367] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.004s
I20240507 08:11:50.766309 5840 raft_consensus.cc:348] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.766593 5840 raft_consensus.cc:374] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.766705 5840 raft_consensus.cc:729] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.767210 5838 ts_tablet_manager.cc:1875] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:50.767292 5901 raft_consensus.cc:482] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.767429 5901 raft_consensus.cc:504] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.768091 5901 leader_election.cc:288] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.768654 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.769232 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.769511 5843 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.769681 5843 ts_tablet_manager.cc:1367] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
W20240507 08:11:50.769992 5378 leader_election.cc:341] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:50.770578 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add94bcf0c844cb78059b7240c7e28c0 consensus state has no leader
I20240507 08:11:50.770505 5843 raft_consensus.cc:348] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:50.770699 5843 raft_consensus.cc:374] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.770928 5843 raft_consensus.cc:729] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.771209 5843 consensus_queue.cc:260] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:50.771638 5377 leader_election.cc:341] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.771817 5377 leader_election.cc:302] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.772891 5901 raft_consensus.cc:2726] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:50.774230 5843 ts_tablet_manager.cc:1398] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:50.774610 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.775288 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.776120 5843 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.777817 5851 raft_consensus.cc:686] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:50.778272 5851 consensus_queue.cc:237] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.779776 5843 tablet_bootstrap.cc:654] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:50.782068 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98880a5ff9604cb4981c033c99e80be7 consensus state has no leader
I20240507 08:11:50.782452 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.783536 5840 consensus_queue.cc:260] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.784407 5840 ts_tablet_manager.cc:1398] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240507 08:11:50.787878 5840 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.788339 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:50.789052 5843 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:50.789208 5843 ts_tablet_manager.cc:1367] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20240507 08:11:50.789566 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:50.789667 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:50.790288 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e1063aa5c7014d84b66f74ce28e3d890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1063aa5c7014d84b66f74ce28e3d890 already in progress: creating tablet
W20240507 08:11:50.790446 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e1063aa5c7014d84b66f74ce28e3d890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1063aa5c7014d84b66f74ce28e3d890 already in progress: creating tablet
I20240507 08:11:50.790920 5843 raft_consensus.cc:348] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:50.791214 5843 raft_consensus.cc:374] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.791970 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:50.792146 5843 raft_consensus.cc:729] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.792539 5843 consensus_queue.cc:260] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:50.793160 5843 ts_tablet_manager.cc:1398] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:50.793552 5843 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:50.795261 5843 tablet_bootstrap.cc:654] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.796468 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5f3654e318b84f969f0a1c0d99221a83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.797345 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f3654e318b84f969f0a1c0d99221a83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.802507 5840 tablet_bootstrap.cc:654] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:50.803223 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6616660bc4a6400a9eb644590ced0e21 consensus state has no leader
I20240507 08:11:50.803736 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e80b1599b03348dfa18d151527ece682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.804378 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e80b1599b03348dfa18d151527ece682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.811671 5840 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.811843 5840 ts_tablet_manager.cc:1367] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240507 08:11:50.812613 5840 raft_consensus.cc:348] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.812853 5840 raft_consensus.cc:374] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.812916 5840 raft_consensus.cc:729] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.813555 5840 consensus_queue.cc:260] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.814733 5840 ts_tablet_manager.cc:1398] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:50.815420 5840 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.817540 5840 tablet_bootstrap.cc:654] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:50.820386 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc619db3a2944ecc9f0112b28de148a6 consensus state has no leader
I20240507 08:11:50.820950 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_31ed6dfebe17497d9a90b837f72f1c2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.821734 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31ed6dfebe17497d9a90b837f72f1c2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.839097 5840 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.839257 5840 ts_tablet_manager.cc:1367] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.006s
I20240507 08:11:50.840034 5840 raft_consensus.cc:348] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.840246 5840 raft_consensus.cc:374] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.840322 5840 raft_consensus.cc:729] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.840559 5840 consensus_queue.cc:260] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.841101 5840 ts_tablet_manager.cc:1398] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:50.841392 5840 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.843078 5840 tablet_bootstrap.cc:654] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.848973 5848 raft_consensus.cc:482] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.849151 5848 raft_consensus.cc:504] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.849834 5848 leader_election.cc:288] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.850530 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:50.850819 5447 raft_consensus.cc:2454] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
I20240507 08:11:50.850917 5838 ts_tablet_manager.cc:1888] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:50.851095 5838 log.cc:1196] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:50.851229 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.851552 5838 ts_tablet_manager.cc:1909] T e8a046cce1f6467390071746a5e39fd9 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:11:50.852337 5517 leader_election.cc:341] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.852948 5520 leader_election.cc:302] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4
I20240507 08:11:50.853482 5848 raft_consensus.cc:2781] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.853587 5848 raft_consensus.cc:482] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.853684 5848 raft_consensus.cc:3037] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.854454 5840 tablet_bootstrap.cc:492] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.854604 5840 ts_tablet_manager.cc:1367] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240507 08:11:50.854614 5256 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet bff85d08695049ec8f04b0fac7fe1a67
I20240507 08:11:50.854934 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet c68b6c08f00d46779a31f4e9eed99f9f from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:50.855680 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9aa4c69483c04eb1a1c0db1c0555cef1" } modify_external_catalogs: true
I20240507 08:11:50.855854 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9aa4c69483c04eb1a1c0db1c0555cef1" } modify_external_catalogs: true
I20240507 08:11:50.856341 5848 raft_consensus.cc:504] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.856381 5840 raft_consensus.cc:348] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.856575 5840 raft_consensus.cc:374] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.856639 5840 raft_consensus.cc:729] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.856925 5840 consensus_queue.cc:260] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.857431 5840 ts_tablet_manager.cc:1398] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240507 08:11:50.857805 5840 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.858083 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:50.858315 5447 raft_consensus.cc:3037] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.862269 5840 tablet_bootstrap.cc:654] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.865696 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
W20240507 08:11:50.866590 5517 leader_election.cc:341] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.867213 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e8a046cce1f6467390071746a5e39fd9 (table default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]) successfully deleted
W20240507 08:11:50.867574 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e1063aa5c7014d84b66f74ce28e3d890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1063aa5c7014d84b66f74ce28e3d890 already in progress: creating tablet
I20240507 08:11:50.868698 5848 leader_election.cc:288] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.862250 5447 raft_consensus.cc:2454] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:11:50.868897 5851 raft_consensus.cc:482] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.869060 5851 raft_consensus.cc:504] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.869334 5520 leader_election.cc:302] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4
I20240507 08:11:50.869763 5851 leader_election.cc:288] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:50.870240 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19efea8905cf4e83a523c278b2f16dc7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:50.870833 5654 leader_election.cc:341] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.871408 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19efea8905cf4e83a523c278b2f16dc7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:50.871764 5446 raft_consensus.cc:2454] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:50.872395 5656 leader_election.cc:302] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.873052 5848 raft_consensus.cc:2781] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.873430 5851 raft_consensus.cc:2781] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.873608 5851 raft_consensus.cc:482] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.873792 5851 raft_consensus.cc:3037] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.877763 5851 raft_consensus.cc:504] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.880035 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19efea8905cf4e83a523c278b2f16dc7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:50.880331 5447 raft_consensus.cc:3037] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.880550 5263 catalog_manager.cc:5484] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:50.881750 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_65d6fd82bd8f4528ad808fb91dd27790 [id=98dc15c6d4b54ec2a8c55a0499349213]). Expected version 4 got 0
I20240507 08:11:50.883953 5447 raft_consensus.cc:2454] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:50.885259 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19efea8905cf4e83a523c278b2f16dc7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:50.885913 5654 leader_election.cc:341] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.886674 5656 leader_election.cc:302] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.886487 5255 catalog_manager.cc:5484] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:50.887034 5907 raft_consensus.cc:2781] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.887141 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:50.887178 5907 raft_consensus.cc:686] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:50.887501 5907 consensus_queue.cc:237] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.890666 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0402c6efc8412aac0d21a545944073 (DEFAULT_TABLE table=default.Table_6c5e09f35af9471b89b6f760b282b361 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.891130 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0402c6efc8412aac0d21a545944073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.891692 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0402c6efc8412aac0d21a545944073 (DEFAULT_TABLE table=default.Table_6c5e09f35af9471b89b6f760b282b361 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.892194 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0402c6efc8412aac0d21a545944073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.896512 5851 leader_election.cc:288] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:50.897924 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a1158486249b4ac59cd8e9db4042a5d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.898535 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1158486249b4ac59cd8e9db4042a5d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.900660 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0402c6efc8412aac0d21a545944073 (DEFAULT_TABLE table=default.Table_6c5e09f35af9471b89b6f760b282b361 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.901202 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0402c6efc8412aac0d21a545944073. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.908907 5696 tablet_service.cc:1467] Processing CreateTablet for tablet cbf7dd6413fb47108cce25cde80738fe (DEFAULT_TABLE table=default.Table_4c3d879d3da3408ea46e6ae4ab672eca [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.909379 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf7dd6413fb47108cce25cde80738fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.909988 5561 tablet_service.cc:1467] Processing CreateTablet for tablet cbf7dd6413fb47108cce25cde80738fe (DEFAULT_TABLE table=default.Table_4c3d879d3da3408ea46e6ae4ab672eca [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.910566 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf7dd6413fb47108cce25cde80738fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.913462 5848 raft_consensus.cc:686] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:11:50.914242 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_be363789908942238275223f8ad4c27f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:50.914841 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be363789908942238275223f8ad4c27f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:50.913770 5848 consensus_queue.cc:237] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.918926 5840 tablet_bootstrap.cc:492] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.919106 5840 ts_tablet_manager.cc:1367] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.061s user 0.007s sys 0.012s
I20240507 08:11:50.920136 5840 raft_consensus.cc:348] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.920394 5840 raft_consensus.cc:374] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.920513 5840 raft_consensus.cc:729] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.921902 5421 tablet_service.cc:1467] Processing CreateTablet for tablet cbf7dd6413fb47108cce25cde80738fe (DEFAULT_TABLE table=default.Table_4c3d879d3da3408ea46e6ae4ab672eca [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.922307 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbf7dd6413fb47108cce25cde80738fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.923395 5840 consensus_queue.cc:260] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.923692 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e49dfd218a24464fbac39a3f4b477462 (DEFAULT_TABLE table=default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.924129 5840 ts_tablet_manager.cc:1398] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:11:50.924163 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49dfd218a24464fbac39a3f4b477462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.925541 5840 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.925922 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 80d82625304942d38b0261f95e235a41 (DEFAULT_TABLE table=default.Table_40c0cbe3de2c4a28b7ac07e4245bb230 [id=478717ae383042c8848a113d90cc85ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.926265 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d82625304942d38b0261f95e235a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.928839 5840 tablet_bootstrap.cc:654] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.930352 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 80d82625304942d38b0261f95e235a41 (DEFAULT_TABLE table=default.Table_40c0cbe3de2c4a28b7ac07e4245bb230 [id=478717ae383042c8848a113d90cc85ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.930768 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d82625304942d38b0261f95e235a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.940642 5840 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.940815 5901 raft_consensus.cc:482] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.940997 5901 raft_consensus.cc:504] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.941661 5901 leader_election.cc:288] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.941818 5840 ts_tablet_manager.cc:1367] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.002s
I20240507 08:11:50.942202 5565 tablet_service.cc:1467] Processing CreateTablet for tablet e49dfd218a24464fbac39a3f4b477462 (DEFAULT_TABLE table=default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.942687 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49dfd218a24464fbac39a3f4b477462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.942806 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:50.943495 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:50.944096 5378 leader_election.cc:341] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:50.944521 5377 leader_election.cc:341] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:50.944626 5377 leader_election.cc:302] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.944070 5840 raft_consensus.cc:348] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.944895 5840 raft_consensus.cc:374] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.945001 5840 raft_consensus.cc:729] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.945205 5901 raft_consensus.cc:2726] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:50.945966 5840 consensus_queue.cc:260] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.947479 5840 ts_tablet_manager.cc:1398] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:11:50.950598 5839 tablet_bootstrap.cc:492] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:50.950749 5839 ts_tablet_manager.cc:1367] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.192s user 0.033s sys 0.059s
I20240507 08:11:50.951565 5839 raft_consensus.cc:348] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.951849 5839 raft_consensus.cc:374] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.951977 5839 raft_consensus.cc:729] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.952834 5839 consensus_queue.cc:260] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.954418 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 80d82625304942d38b0261f95e235a41 (DEFAULT_TABLE table=default.Table_40c0cbe3de2c4a28b7ac07e4245bb230 [id=478717ae383042c8848a113d90cc85ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.954828 5839 ts_tablet_manager.cc:1398] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240507 08:11:50.955771 5420 tablet_service.cc:1467] Processing CreateTablet for tablet e49dfd218a24464fbac39a3f4b477462 (DEFAULT_TABLE table=default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.956270 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49dfd218a24464fbac39a3f4b477462. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.956552 5421 tablet_service.cc:1467] Processing CreateTablet for tablet f29568ebe4774662a8b49b7829cf30e7 (DEFAULT_TABLE table=default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.956866 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29568ebe4774662a8b49b7829cf30e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.959595 5839 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:50.960296 5901 raft_consensus.cc:482] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:50.960517 5901 raft_consensus.cc:504] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.961234 5901 leader_election.cc:288] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.961488 5839 tablet_bootstrap.cc:654] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:50.962478 5378 leader_election.cc:341] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:50.964455 5561 tablet_service.cc:1467] Processing CreateTablet for tablet f29568ebe4774662a8b49b7829cf30e7 (DEFAULT_TABLE table=default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.956746 5697 tablet_service.cc:1467] Processing CreateTablet for tablet f29568ebe4774662a8b49b7829cf30e7 (DEFAULT_TABLE table=default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.957176 5840 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.962082 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a046cce1f6467390071746a5e39fd9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:50.965523 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29568ebe4774662a8b49b7829cf30e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.965487 5719 raft_consensus.cc:2454] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:50.965965 5377 leader_election.cc:302] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.966357 5901 raft_consensus.cc:2781] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:50.966459 5901 raft_consensus.cc:482] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:50.966534 5901 raft_consensus.cc:3037] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.967746 5840 tablet_bootstrap.cc:654] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.969094 5901 raft_consensus.cc:504] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.969864 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f29568ebe4774662a8b49b7829cf30e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:50.970487 5378 leader_election.cc:341] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:50.970800 5901 leader_election.cc:288] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:50.971369 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8a046cce1f6467390071746a5e39fd9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:50.971609 5719 raft_consensus.cc:3037] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:50.974606 5719 raft_consensus.cc:2454] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:50.975118 5377 leader_election.cc:302] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:50.975685 5901 raft_consensus.cc:2781] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:50.975945 5901 raft_consensus.cc:686] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:50.976303 5901 consensus_queue.cc:237] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:50.976765 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d82625304942d38b0261f95e235a41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.981276 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5f3654e318b84f969f0a1c0d99221a83" } modify_external_catalogs: true
I20240507 08:11:50.981527 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5f3654e318b84f969f0a1c0d99221a83" } modify_external_catalogs: true
I20240507 08:11:50.987151 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 697611671ef64c53b0be4f5341b8f270 (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.987624 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697611671ef64c53b0be4f5341b8f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:50.992151 5840 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:50.992347 5840 ts_tablet_manager.cc:1367] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.010s
I20240507 08:11:50.993350 5840 raft_consensus.cc:348] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:50.993724 5840 raft_consensus.cc:374] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:50.993846 5840 raft_consensus.cc:729] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:50.994365 5840 consensus_queue.cc:260] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:50.995062 5840 ts_tablet_manager.cc:1398] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:50.995514 5840 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:50.998200 5840 tablet_bootstrap.cc:654] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:50.998706 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 697611671ef64c53b0be4f5341b8f270 (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:50.999261 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697611671ef64c53b0be4f5341b8f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.005990 5840 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.006214 5840 ts_tablet_manager.cc:1367] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240507 08:11:51.010146 5840 raft_consensus.cc:348] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.010483 5840 raft_consensus.cc:374] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.010581 5840 raft_consensus.cc:729] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.011860 5840 consensus_queue.cc:260] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.012468 5840 ts_tablet_manager.cc:1398] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:51.013631 5840 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.014865 5843 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.015012 5843 ts_tablet_manager.cc:1367] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.222s user 0.069s sys 0.077s
I20240507 08:11:51.014847 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ab41152f87d9489585412dcc24dd2098"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.015470 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab41152f87d9489585412dcc24dd2098 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.015759 5843 raft_consensus.cc:348] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.015884 5840 tablet_bootstrap.cc:654] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.015956 5843 raft_consensus.cc:374] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.016049 5843 raft_consensus.cc:729] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.016295 5843 consensus_queue.cc:260] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.020644 5843 ts_tablet_manager.cc:1398] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:11:51.022018 5840 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.022169 5840 ts_tablet_manager.cc:1367] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240507 08:11:51.022770 5843 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.023692 5840 raft_consensus.cc:348] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.023941 5840 raft_consensus.cc:374] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.024046 5840 raft_consensus.cc:729] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.024256 5840 consensus_queue.cc:260] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.024336 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 697611671ef64c53b0be4f5341b8f270 (DEFAULT_TABLE table=default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.024806 5840 ts_tablet_manager.cc:1398] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:11:51.024834 5843 tablet_bootstrap.cc:654] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.025920 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 697611671ef64c53b0be4f5341b8f270. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.026101 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--webserver_interface=127.4.246.253
--webserver_port=40743
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240507 08:11:51.027911 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_af2b4d29f1e64f2d8e48d5b37c48318a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.028632 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af2b4d29f1e64f2d8e48d5b37c48318a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.034225 5843 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.034412 5843 ts_tablet_manager.cc:1367] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240507 08:11:51.035197 5840 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.035192 5843 raft_consensus.cc:348] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.035431 5843 raft_consensus.cc:374] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.035518 5843 raft_consensus.cc:729] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.035785 5843 consensus_queue.cc:260] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.037168 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 400d2a7b0e224e46a5a580ca22e96eab
I20240507 08:11:51.037873 5843 ts_tablet_manager.cc:1398] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:51.038149 5843 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.038614 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.039080 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:51.039670 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 400d2a7b0e224e46a5a580ca22e96eab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400d2a7b0e224e46a5a580ca22e96eab already in progress: creating tablet
W20240507 08:11:51.039954 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 400d2a7b0e224e46a5a580ca22e96eab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400d2a7b0e224e46a5a580ca22e96eab already in progress: creating tablet
I20240507 08:11:51.040102 5840 tablet_bootstrap.cc:654] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.041962 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.042229 5907 raft_consensus.cc:482] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.042426 5907 raft_consensus.cc:504] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:51.042888 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 400d2a7b0e224e46a5a580ca22e96eab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 400d2a7b0e224e46a5a580ca22e96eab already in progress: creating tablet
I20240507 08:11:51.043272 5907 leader_election.cc:288] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:11:51.044216 5656 leader_election.cc:341] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d6d9ab293113410c857b5de903e21eb5
W20240507 08:11:51.044399 5654 leader_election.cc:341] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:51.044536 5654 leader_election.cc:302] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.044906 5907 raft_consensus.cc:2726] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.045181 5843 tablet_bootstrap.cc:654] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.045253 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 98880a5ff9604cb4981c033c99e80be7 (table default.Table_11f043808d86473e9d1ff0c532d9c4c0 [id=245f6e995eca47f38bdc7495b40a6299]). Expected version 1 got 0
I20240507 08:11:51.045480 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet baa842552d00489998f83e68aa0d516b (table default.Table_b4b47cddd3c949bb9b91f6351807ceff [id=13e03112a7a641828153301cefc1774e]). Expected version 1 got 0
I20240507 08:11:51.048295 5840 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.048858 5840 ts_tablet_manager.cc:1367] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240507 08:11:51.049332 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.049659 5841 tablet_replica.cc:330] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.049901 5841 raft_consensus.cc:2229] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.050112 5841 raft_consensus.cc:2258] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.050918 5840 raft_consensus.cc:348] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.051589 5841 ts_tablet_manager.cc:1875] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.051227 5840 raft_consensus.cc:374] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.052212 5840 raft_consensus.cc:729] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.052831 5840 consensus_queue.cc:260] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.053504 5843 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.053649 5843 ts_tablet_manager.cc:1367] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.001s
I20240507 08:11:51.054440 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.054405 5843 raft_consensus.cc:348] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.054788 5843 raft_consensus.cc:374] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.055029 5843 raft_consensus.cc:729] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.055317 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.055438 5843 consensus_queue.cc:260] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.055938 5840 ts_tablet_manager.cc:1398] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240507 08:11:51.056013 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.056239 5840 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.057407 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.061798 5840 tablet_bootstrap.cc:654] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.062507 5843 ts_tablet_manager.cc:1398] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:11:51.065261 5843 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.066334 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.067209 5843 tablet_bootstrap.cc:654] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.067224 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.068984 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.071354 5840 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.071553 5840 ts_tablet_manager.cc:1367] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
W20240507 08:11:51.072191 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.072548 5840 raft_consensus.cc:348] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.072837 5840 raft_consensus.cc:374] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.072954 5840 raft_consensus.cc:729] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.074401 5841 ts_tablet_manager.cc:1888] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.075666 5840 consensus_queue.cc:260] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.075353 5843 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.076138 5841 log.cc:1196] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/19efea8905cf4e83a523c278b2f16dc7
I20240507 08:11:51.076314 5840 ts_tablet_manager.cc:1398] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240507 08:11:51.077411 5841 ts_tablet_manager.cc:1909] T 19efea8905cf4e83a523c278b2f16dc7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:51.077540 5840 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.076177 5843 ts_tablet_manager.cc:1367] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20240507 08:11:51.078575 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 19efea8905cf4e83a523c278b2f16dc7 (table default.Table_237e57be541d40359403f08fcc681dd8 [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:51.079213 5843 raft_consensus.cc:348] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.079509 5843 raft_consensus.cc:374] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.079633 5843 raft_consensus.cc:729] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.079955 5843 consensus_queue.cc:260] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.081218 5841 tablet_replica.cc:330] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.081559 5841 raft_consensus.cc:2229] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.082417 5841 raft_consensus.cc:2258] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.083022 5841 ts_tablet_manager.cc:1875] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.085273 5843 ts_tablet_manager.cc:1398] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240507 08:11:51.086427 5840 tablet_bootstrap.cc:654] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.086956 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5cab176c4bd34f38b14ac096f79b912e
I20240507 08:11:51.087559 5843 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.087761 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.089082 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5cab176c4bd34f38b14ac096f79b912e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cab176c4bd34f38b14ac096f79b912e already in progress: creating tablet
I20240507 08:11:51.089494 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.090747 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_66645dce4ba84479878b057a6178138a" } new_table_name: "default.Table_41b539df8107441aa4610f0fc94729c2" modify_external_catalogs: true
I20240507 08:11:51.092672 5843 tablet_bootstrap.cc:654] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.093019 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.094432 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:51.095170 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5cab176c4bd34f38b14ac096f79b912e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cab176c4bd34f38b14ac096f79b912e already in progress: creating tablet
I20240507 08:11:51.099872 5907 raft_consensus.cc:482] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.100095 5907 raft_consensus.cc:504] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.100991 5907 leader_election.cc:288] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:11:51.102391 5654 leader_election.cc:341] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: 717460af19164c058f42c7a03095bb30
W20240507 08:11:51.110932 5656 leader_election.cc:341] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 717460af19164c058f42c7a03095bb30
I20240507 08:11:51.111116 5656 leader_election.cc:302] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.111135 5907 raft_consensus.cc:482] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.111346 5907 raft_consensus.cc:504] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.111519 5851 raft_consensus.cc:2726] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.112211 5907 leader_election.cc:288] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:11:51.113675 5654 leader_election.cc:341] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: a485c1dc1cae40fcadce8ce75a5607b2
W20240507 08:11:51.114876 5656 leader_election.cc:341] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: a485c1dc1cae40fcadce8ce75a5607b2
I20240507 08:11:51.115046 5656 leader_election.cc:302] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.115569 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.115617 5907 raft_consensus.cc:2726] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.117836 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 97a43c27b30a4c3cbfd80dce5f28d7c7
I20240507 08:11:51.118389 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 deleted and replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:51.118805 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.120373 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 already in progress: creating tablet
I20240507 08:11:51.124447 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_40c0cbe3de2c4a28b7ac07e4245bb230" } new_table_name: "default.Table_014d10f31cfe4b05a42d8f9c3d3921e3" modify_external_catalogs: true
I20240507 08:11:51.126089 5719 raft_consensus.cc:1261] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:51.126325 5255 catalog_manager.cc:5484] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe673739d8084e7da5d7edd722e857d8 (127.4.246.194). New cstate: current_term: 1 leader_uuid: "fe673739d8084e7da5d7edd722e857d8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:51.126725 5255 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:51.127707 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cdb3e713b1194129bd3f5be0ac3991e6" } new_table_name: "default.Table_a749784e5d4446c4918d32b35efd5451" modify_external_catalogs: true
I20240507 08:11:51.130851 5585 raft_consensus.cc:1261] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:51.132076 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.132274 5901 consensus_queue.cc:1035] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:51.135684 5901 raft_consensus.cc:482] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.135844 5901 raft_consensus.cc:504] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.136025 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.136483 5901 leader_election.cc:288] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.137207 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98880a5ff9604cb4981c033c99e80be7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:51.137490 5720 raft_consensus.cc:2454] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:51.138641 5377 leader_election.cc:302] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.139004 5901 raft_consensus.cc:2781] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.139119 5901 raft_consensus.cc:482] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.139201 5901 raft_consensus.cc:3037] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.140619 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98880a5ff9604cb4981c033c99e80be7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.141256 5260 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 495f7d03a1b94f789a6d82347f443c2d
W20240507 08:11:51.142359 5378 leader_election.cc:341] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.142659 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:51.142170 5901 raft_consensus.cc:504] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.143138 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 34ffe98d424a4eb5a08be067f96a50f9 deleted and replaced by tablet 66184203a41e416e922f6cd532a897e0
I20240507 08:11:51.143752 5901 leader_election.cc:288] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.144268 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 66184203a41e416e922f6cd532a897e0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.144641 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98880a5ff9604cb4981c033c99e80be7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:51.144866 5720 raft_consensus.cc:3037] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.145336 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98880a5ff9604cb4981c033c99e80be7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:51.146173 5378 leader_election.cc:341] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:51.148079 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 already in progress: creating tablet
I20240507 08:11:51.148371 5837 tablet_replica.cc:330] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.148608 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 66184203a41e416e922f6cd532a897e0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.148633 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 66184203a41e416e922f6cd532a897e0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.148710 5837 raft_consensus.cc:2229] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.148886 5838 tablet_replica.cc:330] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.148913 5837 raft_consensus.cc:2258] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.149674 5837 ts_tablet_manager.cc:1875] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.155238 5901 consensus_queue.cc:1035] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:51.156126 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.156673 5843 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:11:51.157205 5377 consensus_peers.cc:485] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:51.157913 5843 ts_tablet_manager.cc:1367] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.003s
I20240507 08:11:51.158406 5720 raft_consensus.cc:2454] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:51.158982 5377 leader_election.cc:302] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.163184 5901 raft_consensus.cc:2781] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.163422 5901 raft_consensus.cc:686] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:51.163763 5901 consensus_queue.cc:237] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.165549 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_38da0b3a42d940728e07debdd163315b" } new_table_name: "default.Table_866be50aa9504f60904fd1dc4d383a17" modify_external_catalogs: true
I20240507 08:11:51.166990 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_11f043808d86473e9d1ff0c532d9c4c0" } modify_external_catalogs: true
I20240507 08:11:51.167165 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_11f043808d86473e9d1ff0c532d9c4c0" } modify_external_catalogs: true
I20240507 08:11:51.168046 5837 ts_tablet_manager.cc:1888] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.168227 5837 log.cc:1196] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:51.158794 5843 raft_consensus.cc:348] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.168334 5843 raft_consensus.cc:374] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.168424 5843 raft_consensus.cc:729] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.168717 5837 ts_tablet_manager.cc:1909] T 34ffe98d424a4eb5a08be067f96a50f9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.169020 5843 consensus_queue.cc:260] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.170022 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_65d6fd82bd8f4528ad808fb91dd27790 [id=98dc15c6d4b54ec2a8c55a0499349213]) successfully deleted
I20240507 08:11:51.172228 5843 ts_tablet_manager.cc:1398] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240507 08:11:51.172643 5843 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.172680 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4e6dbc6b805d4d6888de30dc35f6972b" } new_table_name: "default.Table_b91d41a813624523a849922b73d8bd03" modify_external_catalogs: true
I20240507 08:11:51.174319 5839 tablet_bootstrap.cc:492] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.174432 5840 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.174584 5840 ts_tablet_manager.cc:1367] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.097s user 0.031s sys 0.046s
I20240507 08:11:51.175423 5838 raft_consensus.cc:2229] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.175338 5840 raft_consensus.cc:348] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.175549 5840 raft_consensus.cc:374] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.175619 5840 raft_consensus.cc:729] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.175894 5840 consensus_queue.cc:260] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.176406 5840 ts_tablet_manager.cc:1398] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:51.176733 5840 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.178411 5840 tablet_bootstrap.cc:654] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.179437 5838 pending_rounds.cc:62] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Trying to abort 2 pending ops.
I20240507 08:11:51.179515 5838 pending_rounds.cc:68] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7024924717561483264 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 4 new_table_name: "default.Table_65d6fd82bd8f4528ad808fb91dd27790" dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" new_extra_config { } }
I20240507 08:11:51.179958 5840 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.180087 5840 ts_tablet_manager.cc:1367] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:51.180794 5840 raft_consensus.cc:348] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.180917 5838 pending_rounds.cc:68] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924715648495616 op_type: NO_OP noop_request { }
I20240507 08:11:51.180999 5840 raft_consensus.cc:374] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.181036 5838 raft_consensus.cc:2866] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:51.181084 5840 raft_consensus.cc:729] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.181213 5838 raft_consensus.cc:2258] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.181346 5843 tablet_bootstrap.cc:654] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.181419 5839 ts_tablet_manager.cc:1367] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.215s user 0.031s sys 0.043s
I20240507 08:11:51.181318 5840 consensus_queue.cc:260] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.181910 5840 ts_tablet_manager.cc:1398] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:51.182173 5840 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.182986 5839 raft_consensus.cc:348] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.183236 5839 raft_consensus.cc:374] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.183312 5839 raft_consensus.cc:729] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.183620 5840 tablet_bootstrap.cc:654] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.183542 5839 consensus_queue.cc:260] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.184091 5839 ts_tablet_manager.cc:1398] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:51.184379 5839 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:51.189265 5840 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.189443 5840 ts_tablet_manager.cc:1367] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240507 08:11:51.190348 5907 raft_consensus.cc:482] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.190397 5840 raft_consensus.cc:348] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.190507 5907 raft_consensus.cc:504] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.190616 5840 raft_consensus.cc:374] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.190742 5840 raft_consensus.cc:729] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.190990 5840 consensus_queue.cc:260] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.191272 5907 leader_election.cc:288] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:51.191478 5840 ts_tablet_manager.cc:1398] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:51.191768 5840 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:51.192346 5656 leader_election.cc:341] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5cb62017acbd4472ac5b143eafb36ae9
I20240507 08:11:51.192529 5839 tablet_bootstrap.cc:654] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.192891 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cb62017acbd4472ac5b143eafb36ae9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.193281 5840 tablet_bootstrap.cc:654] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:51.193524 5654 leader_election.cc:341] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.194206 5654 leader_election.cc:302] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:51.194253 5953 consensus_queue.cc:1159] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "fe673739d8084e7da5d7edd722e857d8" status { last_received { term: 1 index: 2 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 2 } }
I20240507 08:11:51.194633 5907 raft_consensus.cc:2726] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.196023 5843 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.196210 5843 ts_tablet_manager.cc:1367] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20240507 08:11:51.197005 5843 raft_consensus.cc:348] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.197237 5843 raft_consensus.cc:374] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.197350 5843 raft_consensus.cc:729] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.197614 5843 consensus_queue.cc:260] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:51.197875 5378 consensus_peers.cc:485] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:51.198228 5907 raft_consensus.cc:482] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.198370 5907 raft_consensus.cc:504] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.198982 5907 leader_election.cc:288] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:11:51.200313 5656 leader_election.cc:341] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 81919c8f720f4c72bc2cc457b095859b
I20240507 08:11:51.199553 5841 ts_tablet_manager.cc:1888] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.201315 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81919c8f720f4c72bc2cc457b095859b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.201969 5654 leader_election.cc:341] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.202098 5654 leader_election.cc:302] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:51.202373 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80d82625304942d38b0261f95e235a41 consensus state has no leader
I20240507 08:11:51.202508 5907 raft_consensus.cc:2726] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.202759 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65d6fd82bd8f4528ad808fb91dd27790" } modify_external_catalogs: true
I20240507 08:11:51.202932 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65d6fd82bd8f4528ad808fb91dd27790" } modify_external_catalogs: true
W20240507 08:11:51.204136 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 consensus state has no leader
I20240507 08:11:51.204532 5840 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.204493 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.204671 5840 ts_tablet_manager.cc:1367] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240507 08:11:51.204777 5907 raft_consensus.cc:482] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.204931 5907 raft_consensus.cc:504] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:51.205024 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.205523 5907 leader_election.cc:288] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:51.205636 5840 raft_consensus.cc:348] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.205824 5840 raft_consensus.cc:374] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.205910 5840 raft_consensus.cc:729] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.206105 5840 consensus_queue.cc:260] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.206598 5840 ts_tablet_manager.cc:1398] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:11:51.206676 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.206893 5840 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.206976 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.207456 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:51.208631 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34e4e33f52df432b8d043171d7232c2a consensus state has no leader
I20240507 08:11:51.208848 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet e0f3ee2b04de4cdf8a621b39f1476577 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.209074 5840 tablet_bootstrap.cc:654] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.202630 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f8abdaf2e2eb424eb4b547eae6405bda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.212051 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8abdaf2e2eb424eb4b547eae6405bda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.215413 5838 ts_tablet_manager.cc:1875] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.200711 5843 ts_tablet_manager.cc:1398] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240507 08:11:51.216002 5843 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.210582 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc619db3a2944ecc9f0112b28de148a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.217105 5447 raft_consensus.cc:2379] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cbb887175f345ada2433adaec906fd4 in current term 1: Already voted for candidate fe673739d8084e7da5d7edd722e857d8 in this term.
I20240507 08:11:51.218080 5656 leader_election.cc:302] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.218619 5907 raft_consensus.cc:3037] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.218724 5843 tablet_bootstrap.cc:654] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.223278 5841 log.cc:1196] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:51.225472 5841 ts_tablet_manager.cc:1909] T 71bbe28a013344bcb6529247d7139ab9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:51.227320 5907 raft_consensus.cc:2726] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:51.232940 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1cf2df0d3bd46268e22713fcc135a74 consensus state has no leader
I20240507 08:11:51.234714 5839 tablet_bootstrap.cc:492] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.234939 5839 ts_tablet_manager.cc:1367] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.001s
I20240507 08:11:51.236101 5839 raft_consensus.cc:348] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.236399 5839 raft_consensus.cc:374] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.236474 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 71bbe28a013344bcb6529247d7139ab9 (table default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]) successfully deleted
I20240507 08:11:51.236513 5839 raft_consensus.cc:729] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.236902 5839 consensus_queue.cc:260] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.237159 5841 tablet_replica.cc:330] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.239781 5839 ts_tablet_manager.cc:1398] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240507 08:11:51.241441 5843 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.241655 5843 ts_tablet_manager.cc:1367] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.004s
I20240507 08:11:51.242173 5839 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:51.242482 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4899c088531480ea15364af94595ee5 consensus state has no leader
I20240507 08:11:51.242754 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet e1cf2df0d3bd46268e22713fcc135a74 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.243393 5838 ts_tablet_manager.cc:1888] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:51.243309 5843 raft_consensus.cc:348] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.243497 5843 raft_consensus.cc:374] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.243530 5838 log.cc:1196] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:51.243582 5843 raft_consensus.cc:729] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.243839 5843 consensus_queue.cc:260] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.244057 5838 ts_tablet_manager.cc:1909] T 34ffe98d424a4eb5a08be067f96a50f9 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:51.240831 5841 raft_consensus.cc:2229] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.246747 5841 raft_consensus.cc:2258] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.246047 5839 tablet_bootstrap.cc:654] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.247371 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 34ffe98d424a4eb5a08be067f96a50f9 (table default.Table_65d6fd82bd8f4528ad808fb91dd27790 [id=98dc15c6d4b54ec2a8c55a0499349213]) successfully deleted
I20240507 08:11:51.252264 5843 ts_tablet_manager.cc:1398] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:11:51.253844 5839 tablet_bootstrap.cc:492] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.253880 5843 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.254206 5839 ts_tablet_manager.cc:1367] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240507 08:11:51.255183 5839 raft_consensus.cc:348] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.255540 5839 raft_consensus.cc:374] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.255622 5839 raft_consensus.cc:729] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.255959 5839 consensus_queue.cc:260] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.257464 5843 tablet_bootstrap.cc:654] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.259778 5960 tablet.cc:1722] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:51.262638 5839 ts_tablet_manager.cc:1398] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:11:51.262284 5841 ts_tablet_manager.cc:1875] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.266497 5839 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:51.268957 5839 tablet_bootstrap.cc:654] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.271272 5901 raft_consensus.cc:482] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.271596 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 81919c8f720f4c72bc2cc457b095859b from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.271411 5901 raft_consensus.cc:504] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.272773 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.273517 5378 leader_election.cc:341] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.274001 5901 leader_election.cc:288] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.275249 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:51.276074 5377 leader_election.cc:341] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.276283 5377 leader_election.cc:302] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.276582 5901 raft_consensus.cc:2726] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.280403 5843 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.280925 5843 ts_tablet_manager.cc:1367] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20240507 08:11:51.281912 5843 raft_consensus.cc:348] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.282181 5843 raft_consensus.cc:374] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.282303 5843 raft_consensus.cc:729] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.282603 5843 consensus_queue.cc:260] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.283023 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f956db8e0c684a60a84296ba0504178d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.283313 5843 ts_tablet_manager.cc:1398] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240507 08:11:51.283600 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f956db8e0c684a60a84296ba0504178d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.288574 5843 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.294642 5840 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.294906 5840 ts_tablet_manager.cc:1367] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.088s user 0.032s sys 0.047s
I20240507 08:11:51.295776 5840 raft_consensus.cc:348] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.296160 5840 raft_consensus.cc:374] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.296343 5840 raft_consensus.cc:729] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.296677 5840 consensus_queue.cc:260] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.297317 5840 ts_tablet_manager.cc:1398] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:51.299669 5944 mvcc.cc:205] Tried to move back new op lower bound from 7024924717561483264 to 7024924715648495616. Current Snapshot: MvccSnapshot[applied={T|T < 7024924717561483264}]
I20240507 08:11:51.300925 5839 tablet_bootstrap.cc:492] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.301077 5839 ts_tablet_manager.cc:1367] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.009s
I20240507 08:11:51.301170 5843 tablet_bootstrap.cc:654] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.301879 5839 raft_consensus.cc:348] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.302079 5839 raft_consensus.cc:374] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.302166 5839 raft_consensus.cc:729] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.302436 5839 consensus_queue.cc:260] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.304777 5839 ts_tablet_manager.cc:1398] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:51.306624 5839 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:51.308096 5839 tablet_bootstrap.cc:654] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.310428 5840 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.310797 5907 raft_consensus.cc:482] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.311048 5907 raft_consensus.cc:504] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.311812 5907 leader_election.cc:288] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.312218 5840 tablet_bootstrap.cc:654] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.312341 5843 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.312513 5843 ts_tablet_manager.cc:1367] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.005s
I20240507 08:11:51.312582 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5220905954dc9b078e00e0db69fd2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.312822 5447 raft_consensus.cc:2454] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.312984 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5220905954dc9b078e00e0db69fd2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.313310 5843 raft_consensus.cc:348] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.314497 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e0f3ee2b04de4cdf8a621b39f1476577
I20240507 08:11:51.315126 5256 catalog_manager.cc:6554] ReplaceTablet: tablet e0f3ee2b04de4cdf8a621b39f1476577 deleted and replaced by tablet 270a1fd15925416483c211021549315c
W20240507 08:11:51.315582 5654 leader_election.cc:341] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.315589 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0c7582240f584dd8a2969c426b8aeb25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.316076 5656 leader_election.cc:302] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:51.316114 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c7582240f584dd8a2969c426b8aeb25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.316560 5907 raft_consensus.cc:2781] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.316649 5907 raft_consensus.cc:482] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.316735 5907 raft_consensus.cc:3037] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.318006 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 with delete_type TABLET_DATA_DELETED (replaced by tablet 270a1fd15925416483c211021549315c) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.318748 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e0f3ee2b04de4cdf8a621b39f1476577 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f3ee2b04de4cdf8a621b39f1476577 already in progress: creating tablet
I20240507 08:11:51.319196 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_be363789908942238275223f8ad4c27f" } modify_external_catalogs: true
I20240507 08:11:51.319375 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_be363789908942238275223f8ad4c27f" } modify_external_catalogs: true
I20240507 08:11:51.318070 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 with delete_type TABLET_DATA_DELETED (replaced by tablet 270a1fd15925416483c211021549315c) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.320662 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 with delete_type TABLET_DATA_DELETED (replaced by tablet 270a1fd15925416483c211021549315c) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.320721 5843 raft_consensus.cc:374] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.320837 5843 raft_consensus.cc:729] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.321094 5843 consensus_queue.cc:260] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.323995 5841 ts_tablet_manager.cc:1888] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.324103 5841 log.cc:1196] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/2cfa9a247eba402cb3f0b818ad9f837a
I20240507 08:11:51.324481 5841 ts_tablet_manager.cc:1909] T 2cfa9a247eba402cb3f0b818ad9f837a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:51.325381 5841 tablet_replica.cc:330] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.325488 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 2cfa9a247eba402cb3f0b818ad9f837a (table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]) successfully deleted
I20240507 08:11:51.325752 5843 ts_tablet_manager.cc:1398] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240507 08:11:51.326020 5907 raft_consensus.cc:504] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.326462 5841 raft_consensus.cc:2229] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.326623 5841 raft_consensus.cc:2258] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.326973 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5220905954dc9b078e00e0db69fd2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.327136 5841 ts_tablet_manager.cc:1875] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:51.327816 5654 leader_election.cc:341] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:51.328189 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e0f3ee2b04de4cdf8a621b39f1476577 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e0f3ee2b04de4cdf8a621b39f1476577 already in progress: creating tablet
I20240507 08:11:51.329384 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ea5220905954dc9b078e00e0db69fd2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:51.329608 5447 raft_consensus.cc:3037] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.329892 5843 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.331573 5843 tablet_bootstrap.cc:654] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.332376 5447 raft_consensus.cc:2454] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:51.333415 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e1cf2df0d3bd46268e22713fcc135a74
I20240507 08:11:51.333902 5259 catalog_manager.cc:6554] ReplaceTablet: tablet e1cf2df0d3bd46268e22713fcc135a74 deleted and replaced by tablet f49dd4772f8e46478638b29c009c83ea
I20240507 08:11:51.333914 5656 leader_election.cc:302] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.334210 5851 raft_consensus.cc:2781] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.334367 5851 raft_consensus.cc:686] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:51.334650 5851 consensus_queue.cc:237] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.337991 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 with delete_type TABLET_DATA_DELETED (replaced by tablet f49dd4772f8e46478638b29c009c83ea) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.338827 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 with delete_type TABLET_DATA_DELETED (replaced by tablet f49dd4772f8e46478638b29c009c83ea) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.339709 5907 leader_election.cc:288] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:51.339700 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e1cf2df0d3bd46268e22713fcc135a74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1cf2df0d3bd46268e22713fcc135a74 already in progress: creating tablet
I20240507 08:11:51.343186 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5ee87936fa1d4b14a95e89fcb6b0b4b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.343943 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ee87936fa1d4b14a95e89fcb6b0b4b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.345748 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 91c6cb7920df47aa97be6555c587798a from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.346613 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 with delete_type TABLET_DATA_DELETED (replaced by tablet f49dd4772f8e46478638b29c009c83ea) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.347219 5841 ts_tablet_manager.cc:1888] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:11:51.348045 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e1cf2df0d3bd46268e22713fcc135a74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1cf2df0d3bd46268e22713fcc135a74 already in progress: creating tablet
I20240507 08:11:51.348707 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2f5246f5a6874c848d060c15dd9be10f" } modify_external_catalogs: true
I20240507 08:11:51.348908 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2f5246f5a6874c848d060c15dd9be10f" } modify_external_catalogs: true
I20240507 08:11:51.352013 5262 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a177fb63333545ae9120acb5df9be243
I20240507 08:11:51.352391 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fea6e34b279d43b2918afeac8c53ceec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.352872 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fea6e34b279d43b2918afeac8c53ceec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.358598 5841 log.cc:1196] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/120ebfd34a9e4d73a215cb9206013728
I20240507 08:11:51.359388 5841 ts_tablet_manager.cc:1909] T 120ebfd34a9e4d73a215cb9206013728 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:51.359396 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d3d5d2e042244132918d76affc48a306"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.359957 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3d5d2e042244132918d76affc48a306 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.361018 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 120ebfd34a9e4d73a215cb9206013728 (table default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]) successfully deleted
I20240507 08:11:51.362149 5841 tablet_replica.cc:330] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.363727 5841 raft_consensus.cc:2229] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.364362 5841 raft_consensus.cc:2258] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.365805 5841 ts_tablet_manager.cc:1875] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.366847 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a10d53fdc3bc4f009010ffe48156725d" } modify_external_catalogs: true
I20240507 08:11:51.367035 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a10d53fdc3bc4f009010ffe48156725d" } modify_external_catalogs: true
I20240507 08:11:51.368155 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:51.368872 5264 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 66184203a41e416e922f6cd532a897e0
I20240507 08:11:51.369112 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ea5220905954dc9b078e00e0db69fd2 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.369362 5421 tablet_service.cc:1467] Processing CreateTablet for tablet cb7acbdb357445adb9ce69103c782a7f (DEFAULT_TABLE table=default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.369796 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7acbdb357445adb9ce69103c782a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.375157 5423 tablet_service.cc:1467] Processing CreateTablet for tablet ea17889619fa420fbeacf1b41594c598 (DEFAULT_TABLE table=default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.375427 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e (DEFAULT_TABLE table=default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.375492 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:51.375561 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea17889619fa420fbeacf1b41594c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.375492 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 344bf4ab39f74accb383d695578de604 (DEFAULT_TABLE table=default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.375759 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972bc0a5ba924e0fb946cdacd9575c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.375816 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 98880a5ff9604cb4981c033c99e80be7 (table default.Table_11f043808d86473e9d1ff0c532d9c4c0 [id=245f6e995eca47f38bdc7495b40a6299]). Expected version 1 got 0
I20240507 08:11:51.375942 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344bf4ab39f74accb383d695578de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.375998 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:51.376153 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet baa842552d00489998f83e68aa0d516b (table default.Table_b4b47cddd3c949bb9b91f6351807ceff [id=13e03112a7a641828153301cefc1774e]). Expected version 1 got 0
I20240507 08:11:51.376231 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba24c078af9472da06793c58f147290 (DEFAULT_TABLE table=default.Table_e97ccca6ca324143a7e713d916068114 [id=8ea36271c6814ba58435556d3ff59f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.376574 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba24c078af9472da06793c58f147290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.377907 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 67c8624285d946e597227430a5714e2f (DEFAULT_TABLE table=default.Table_4ea4bbd2ff0a49a89be1371511b5986c [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.378284 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c8624285d946e597227430a5714e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.379446 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e (DEFAULT_TABLE table=default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.379878 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972bc0a5ba924e0fb946cdacd9575c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.380249 5565 tablet_service.cc:1467] Processing CreateTablet for tablet cb7acbdb357445adb9ce69103c782a7f (DEFAULT_TABLE table=default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.380631 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7acbdb357445adb9ce69103c782a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.381489 5427 tablet_service.cc:1467] Processing CreateTablet for tablet e4c064b969844d938fddfe7f043c0b92 (DEFAULT_TABLE table=default.Table_431fe83bbf04464e84771e65abee1c42 [id=2435a4b9c9a7464aafa19339bf7c9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.381920 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e (DEFAULT_TABLE table=default.Table_31ed6dfebe17497d9a90b837f72f1c2e [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.381855 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 67c8624285d946e597227430a5714e2f (DEFAULT_TABLE table=default.Table_4ea4bbd2ff0a49a89be1371511b5986c [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.382225 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c8624285d946e597227430a5714e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.382377 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba24c078af9472da06793c58f147290 (DEFAULT_TABLE table=default.Table_e97ccca6ca324143a7e713d916068114 [id=8ea36271c6814ba58435556d3ff59f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.382709 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba24c078af9472da06793c58f147290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.385725 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cfe3c25a9a84eab9091502955453951 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.385882 5558 tablet_service.cc:1467] Processing CreateTablet for tablet ea17889619fa420fbeacf1b41594c598 (DEFAULT_TABLE table=default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.386014 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 34ffe98d424a4eb5a08be067f96a50f9 with delete_type TABLET_DATA_DELETED (replaced by tablet 66184203a41e416e922f6cd532a897e0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.386075 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 344bf4ab39f74accb383d695578de604 (DEFAULT_TABLE table=default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.386860 5838 tablet_replica.cc:330] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.386924 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea17889619fa420fbeacf1b41594c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.387157 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344bf4ab39f74accb383d695578de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.387980 5838 raft_consensus.cc:2229] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.387950 5563 tablet_service.cc:1467] Processing CreateTablet for tablet e4c064b969844d938fddfe7f043c0b92 (DEFAULT_TABLE table=default.Table_431fe83bbf04464e84771e65abee1c42 [id=2435a4b9c9a7464aafa19339bf7c9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.388172 5838 raft_consensus.cc:2258] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.388283 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c064b969844d938fddfe7f043c0b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.389045 5838 ts_tablet_manager.cc:1875] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.389688 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e (DEFAULT_TABLE table=default.Table_31ed6dfebe17497d9a90b837f72f1c2e [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.390187 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca8fc2c7f3849c380fea56862ffc26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.391865 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 9ba24c078af9472da06793c58f147290 (DEFAULT_TABLE table=default.Table_e97ccca6ca324143a7e713d916068114 [id=8ea36271c6814ba58435556d3ff59f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.392273 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ba24c078af9472da06793c58f147290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.392928 5696 tablet_service.cc:1467] Processing CreateTablet for tablet cb7acbdb357445adb9ce69103c782a7f (DEFAULT_TABLE table=default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.393339 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb7acbdb357445adb9ce69103c782a7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.393864 5700 tablet_service.cc:1467] Processing CreateTablet for tablet ea17889619fa420fbeacf1b41594c598 (DEFAULT_TABLE table=default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.394245 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea17889619fa420fbeacf1b41594c598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.397876 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 344bf4ab39f74accb383d695578de604 (DEFAULT_TABLE table=default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.398170 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 67c8624285d946e597227430a5714e2f (DEFAULT_TABLE table=default.Table_4ea4bbd2ff0a49a89be1371511b5986c [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.398201 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4557db659d1e4591a2bffcc5077e92dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.398439 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 344bf4ab39f74accb383d695578de604. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.398656 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67c8624285d946e597227430a5714e2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:51.398785 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4557db659d1e4591a2bffcc5077e92dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.397876 5699 tablet_service.cc:1467] Processing CreateTablet for tablet e4c064b969844d938fddfe7f043c0b92 (DEFAULT_TABLE table=default.Table_431fe83bbf04464e84771e65abee1c42 [id=2435a4b9c9a7464aafa19339bf7c9277]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.399153 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c064b969844d938fddfe7f043c0b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.399292 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e (DEFAULT_TABLE table=default.Table_31ed6dfebe17497d9a90b837f72f1c2e [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.397850 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e (DEFAULT_TABLE table=default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.399627 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca8fc2c7f3849c380fea56862ffc26e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.399793 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 972bc0a5ba924e0fb946cdacd9575c1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.400169 5851 raft_consensus.cc:482] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.400300 5851 raft_consensus.cc:504] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.400995 5851 leader_election.cc:288] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.401671 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0074cc70e248ac8b5e90cf57b441c3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.401921 5447 raft_consensus.cc:2454] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.402360 5656 leader_election.cc:302] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.403144 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0074cc70e248ac8b5e90cf57b441c3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.403688 5654 leader_election.cc:341] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.404018 5851 raft_consensus.cc:2781] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.404122 5851 raft_consensus.cc:482] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.404208 5851 raft_consensus.cc:3037] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.404470 5838 ts_tablet_manager.cc:1888] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.408377 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30a40b16a51c4ad28c5fee73ed5e4185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.409058 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_745f979f496a402fb4868ac8ed7099ff" } new_table_name: "default.Table_eff2f49d93764317a595c308278c0732" modify_external_catalogs: true
W20240507 08:11:51.409526 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30a40b16a51c4ad28c5fee73ed5e4185 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.415195 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 91c6cb7920df47aa97be6555c587798a
I20240507 08:11:51.415828 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 91c6cb7920df47aa97be6555c587798a deleted and replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:11:51.416004 5907 raft_consensus.cc:482] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.416154 5907 raft_consensus.cc:504] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.416759 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ea17889619fa420fbeacf1b41594c598
I20240507 08:11:51.416900 5907 leader_election.cc:288] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:51.417703 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.418071 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:51.418397 5656 leader_election.cc:341] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 71bbe28a013344bcb6529247d7139ab9
W20240507 08:11:51.418519 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 91c6cb7920df47aa97be6555c587798a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c6cb7920df47aa97be6555c587798a already in progress: creating tablet
I20240507 08:11:51.418597 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.419181 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ea17889619fa420fbeacf1b41594c598 because the tablet was not found. No further retry: Not found: Tablet not found: ea17889619fa420fbeacf1b41594c598
I20240507 08:11:51.421341 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f8abdaf2e2eb424eb4b547eae6405bda" } new_table_name: "default.Table_e5a3509f77064f24a47bc7d69395d0c7" modify_external_catalogs: true
I20240507 08:11:51.425524 5851 raft_consensus.cc:504] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.426328 5907 raft_consensus.cc:482] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.426479 5907 raft_consensus.cc:504] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.426868 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0074cc70e248ac8b5e90cf57b441c3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:51.427094 5447 raft_consensus.cc:3037] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.427134 5907 leader_election.cc:288] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.429636 5851 leader_election.cc:288] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.429747 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4c064b969844d938fddfe7f043c0b92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.430290 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca8fc2c7f3849c380fea56862ffc26e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:51.430763 5656 leader_election.cc:341] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 120ebfd34a9e4d73a215cb9206013728
I20240507 08:11:51.431779 5447 raft_consensus.cc:2454] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:51.432274 5656 leader_election.cc:302] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.433177 5840 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.433310 5840 ts_tablet_manager.cc:1367] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.123s user 0.010s sys 0.006s
I20240507 08:11:51.433856 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b4b47cddd3c949bb9b91f6351807ceff" } new_table_name: "default.Table_5fc65aee40154e6296dc8b3f3076a4f3" modify_external_catalogs: true
I20240507 08:11:51.434059 5840 raft_consensus.cc:348] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.434270 5840 raft_consensus.cc:374] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.434350 5840 raft_consensus.cc:729] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.435590 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f29568ebe4774662a8b49b7829cf30e7
I20240507 08:11:51.436782 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5ee87936fa1d4b14a95e89fcb6b0b4b1" } new_table_name: "default.Table_5aa2ec19328a431f8277539512fac60e" modify_external_catalogs: true
I20240507 08:11:51.437439 5851 raft_consensus.cc:2781] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.437621 5851 raft_consensus.cc:686] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:51.437711 5838 log.cc:1196] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5cfe3c25a9a84eab9091502955453951
I20240507 08:11:51.437902 5851 consensus_queue.cc:237] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.438283 5901 raft_consensus.cc:482] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.438424 5901 raft_consensus.cc:504] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.439085 5901 leader_election.cc:288] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.440371 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708d4bee30cc45bd9b86c56beb2c3ece" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:51.440582 5720 raft_consensus.cc:2454] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:51.440999 5377 leader_election.cc:302] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.441263 5901 raft_consensus.cc:2781] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.441377 5901 raft_consensus.cc:482] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.441455 5901 raft_consensus.cc:3037] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:51.442948 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5228590c8e7d4c34b3acab9eb16665d3 consensus state has no leader
I20240507 08:11:51.444538 5901 raft_consensus.cc:504] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.445736 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.445791 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e3e2810edd06425882e48b9aeaada4f6" } new_table_name: "default.Table_3c344b571115426b99a5e43c1654596c" modify_external_catalogs: true
I20240507 08:11:51.446208 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708d4bee30cc45bd9b86c56beb2c3ece" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:51.446382 5720 raft_consensus.cc:3037] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:51.446771 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f29568ebe4774662a8b49b7829cf30e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f29568ebe4774662a8b49b7829cf30e7 already in progress: creating tablet
I20240507 08:11:51.447110 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.449080 5720 raft_consensus.cc:2454] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:51.449872 5901 leader_election.cc:288] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.450542 5377 leader_election.cc:302] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.450950 5840 consensus_queue.cc:260] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:51.458050 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0ea74702c4b4514a31f0fad7e7a2f5b has no consensus state
I20240507 08:11:51.462332 5840 ts_tablet_manager.cc:1398] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240507 08:11:51.462643 5840 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.464943 5840 tablet_bootstrap.cc:654] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.465232 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 98880a5ff9604cb4981c033c99e80be7
I20240507 08:11:51.467649 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 98880a5ff9604cb4981c033c99e80be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:51.468065 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet baa842552d00489998f83e68aa0d516b consensus state has no leader
W20240507 08:11:51.468145 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet add94bcf0c844cb78059b7240c7e28c0 (table default.Table_237e57be541d40359403f08fcc681dd8 [id=0a13053ee3a44ee1a9588eb3ba56b74f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet add94bcf0c844cb78059b7240c7e28c0 consensus state has no leader
I20240507 08:11:51.468587 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 98880a5ff9604cb4981c033c99e80be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:51.469388 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e3ffaf1781e4b108f305346476905be has no consensus state
W20240507 08:11:51.475633 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa932c6ad48f4ac183aabba8699cb148 consensus state has no leader
I20240507 08:11:51.477305 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet e8a046cce1f6467390071746a5e39fd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.477567 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.477758 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.477842 5837 tablet_replica.cc:330] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.478029 5837 raft_consensus.cc:2229] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.478214 5837 raft_consensus.cc:2258] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.478765 5837 ts_tablet_manager.cc:1875] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.479022 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.479351 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.479625 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.479840 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.480167 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bbe28a013344bcb6529247d7139ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.480388 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 717460af19164c058f42c7a03095bb30 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.481884 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d9f6ed618e184cc6aece7c2d0dca9df8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.482105 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 81919c8f720f4c72bc2cc457b095859b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.482355 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet d6d9ab293113410c857b5de903e21eb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.482595 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f6ed618e184cc6aece7c2d0dca9df8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.485885 5901 raft_consensus.cc:2781] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.486066 5901 raft_consensus.cc:686] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:51.486353 5901 consensus_queue.cc:237] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.487044 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.487478 5837 ts_tablet_manager.cc:1888] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.487663 5837 log.cc:1196] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:51.488176 5837 ts_tablet_manager.cc:1909] T e8a046cce1f6467390071746a5e39fd9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.488919 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_914ee05f51f14d84bfc1418f5a9cfb27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.489236 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e8a046cce1f6467390071746a5e39fd9 (table default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]) successfully deleted
I20240507 08:11:51.489641 5837 tablet_replica.cc:330] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.489951 5837 raft_consensus.cc:2229] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:51.489943 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_914ee05f51f14d84bfc1418f5a9cfb27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.490195 5837 raft_consensus.cc:2258] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.490841 5837 ts_tablet_manager.cc:1875] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.490830 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.491652 5840 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.491830 5840 ts_tablet_manager.cc:1367] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20240507 08:11:51.493944 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.495643 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 19efea8905cf4e83a523c278b2f16dc7 with delete_type TABLET_DATA_DELETED (replaced by tablet add94bcf0c844cb78059b7240c7e28c0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.497722 5840 raft_consensus.cc:348] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.497984 5840 raft_consensus.cc:374] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.498068 5840 raft_consensus.cc:729] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.500416 5840 consensus_queue.cc:260] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.504191 5837 ts_tablet_manager.cc:1888] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.504341 5837 log.cc:1196] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:51.504465 5840 ts_tablet_manager.cc:1398] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240507 08:11:51.504688 5837 ts_tablet_manager.cc:1909] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.506343 5843 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.506525 5843 ts_tablet_manager.cc:1367] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.177s user 0.008s sys 0.021s
I20240507 08:11:51.507052 5840 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.507404 5843 raft_consensus.cc:348] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.507647 5843 raft_consensus.cc:374] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.507733 5843 raft_consensus.cc:729] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.507998 5843 consensus_queue.cc:260] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.508543 5843 ts_tablet_manager.cc:1398] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:51.508572 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (table default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]) successfully deleted
I20240507 08:11:51.508653 5840 tablet_bootstrap.cc:654] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.508883 5843 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.509286 5837 tablet_replica.cc:330] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.510005 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet a485c1dc1cae40fcadce8ce75a5607b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.510005 5837 raft_consensus.cc:2229] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.510596 5843 tablet_bootstrap.cc:654] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.511440 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 98880a5ff9604cb4981c033c99e80be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.512231 5837 raft_consensus.cc:2258] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.514067 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.514406 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.518095 5837 ts_tablet_manager.cc:1875] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.518617 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.518958 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.520218 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.520642 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7004d9f9040b4d57abb409a848b45e48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.521562 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7004d9f9040b4d57abb409a848b45e48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.524242 5841 ts_tablet_manager.cc:1888] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.524366 5841 log.cc:1196] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:51.524704 5841 ts_tablet_manager.cc:1909] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:51.525908 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 81919c8f720f4c72bc2cc457b095859b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.525878 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed0074cc70e248ac8b5e90cf57b441c3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.526139 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 98880a5ff9604cb4981c033c99e80be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.525770 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (table default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]) successfully deleted
I20240507 08:11:51.531942 5837 ts_tablet_manager.cc:1888] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.533843 5837 log.cc:1196] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5cb62017acbd4472ac5b143eafb36ae9
I20240507 08:11:51.534299 5837 ts_tablet_manager.cc:1909] T 5cb62017acbd4472ac5b143eafb36ae9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.534759 5838 ts_tablet_manager.cc:1909] T 5cfe3c25a9a84eab9091502955453951 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:51.535671 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.538931 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5cb62017acbd4472ac5b143eafb36ae9 (table default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]) successfully deleted
I20240507 08:11:51.539261 5837 tablet_replica.cc:330] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.539394 5848 raft_consensus.cc:482] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.539520 5837 raft_consensus.cc:2229] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.539532 5848 raft_consensus.cc:504] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.539685 5837 raft_consensus.cc:2258] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.540344 5837 ts_tablet_manager.cc:1875] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:51.541378 5654 leader_election.cc:341] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Not found: Tablet not found: 71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:51.541535 5654 leader_election.cc:302] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:51.541980 5654 leader_election.cc:341] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.542281 5907 raft_consensus.cc:2726] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.545459 5838 tablet_replica.cc:330] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.545667 5838 raft_consensus.cc:2229] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.547072 5838 raft_consensus.cc:2258] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.546113 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "120ebfd34a9e4d73a215cb9206013728" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.547989 5654 leader_election.cc:341] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.548108 5654 leader_election.cc:302] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.548542 5907 raft_consensus.cc:2726] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:51.548782 5848 leader_election.cc:288] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.551856 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_06c4bdeddaf4480f82cec8f330889325"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.552615 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06c4bdeddaf4480f82cec8f330889325 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.554294 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b67c0c470e744311924705a8842f1926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.555434 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e80b1599b03348dfa18d151527ece682" } modify_external_catalogs: true
I20240507 08:11:51.555652 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e80b1599b03348dfa18d151527ece682" } modify_external_catalogs: true
I20240507 08:11:51.556062 5837 ts_tablet_manager.cc:1888] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.556243 5837 log.cc:1196] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/2cfa9a247eba402cb3f0b818ad9f837a
W20240507 08:11:51.556438 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b67c0c470e744311924705a8842f1926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.556615 5837 ts_tablet_manager.cc:1909] T 2cfa9a247eba402cb3f0b818ad9f837a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.556790 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cb62017acbd4472ac5b143eafb36ae9 with delete_type TABLET_DATA_DELETED (replaced by tablet 1d16af3fd2a6448a82146edbb2362715) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.557565 5837 tablet_replica.cc:330] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.557708 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 2cfa9a247eba402cb3f0b818ad9f837a (table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]) successfully deleted
I20240507 08:11:51.557946 5837 raft_consensus.cc:2229] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.558271 5837 raft_consensus.cc:2258] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.558261 5838 ts_tablet_manager.cc:1875] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.560568 5837 ts_tablet_manager.cc:1875] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.561293 5839 tablet_bootstrap.cc:492] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.562086 5839 ts_tablet_manager.cc:1367] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.255s user 0.012s sys 0.009s
I20240507 08:11:51.563263 5839 raft_consensus.cc:348] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.564100 5839 raft_consensus.cc:374] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.564194 5839 raft_consensus.cc:729] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.564321 5907 raft_consensus.cc:482] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.564494 5907 raft_consensus.cc:504] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.564637 5839 consensus_queue.cc:260] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.565177 5839 ts_tablet_manager.cc:1398] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:51.565236 5907 leader_election.cc:288] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:51.566634 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_13e0fb6057f044939108be222e8cb0f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.567291 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13e0fb6057f044939108be222e8cb0f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.572398 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_923d5aad72744e84a98819df252a2e2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.573223 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_923d5aad72744e84a98819df252a2e2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.573335 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.573565 5585 raft_consensus.cc:2454] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.574343 5907 raft_consensus.cc:482] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.574488 5907 raft_consensus.cc:504] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.574605 5654 leader_election.cc:302] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:11:51.574894 5891 raft_consensus.cc:2781] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.575004 5891 raft_consensus.cc:482] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.575107 5891 raft_consensus.cc:3037] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.575248 5907 leader_election.cc:288] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.577162 5837 ts_tablet_manager.cc:1888] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.578680 5837 log.cc:1196] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/120ebfd34a9e4d73a215cb9206013728
I20240507 08:11:51.579103 5837 ts_tablet_manager.cc:1909] T 120ebfd34a9e4d73a215cb9206013728 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.579639 5891 raft_consensus.cc:504] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.580287 5891 leader_election.cc:288] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:51.580613 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400d2a7b0e224e46a5a580ca22e96eab" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.581171 5654 leader_election.cc:341] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.581456 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.582012 5839 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:51.582048 5585 raft_consensus.cc:3037] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.583611 5837 tablet_replica.cc:330] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.583698 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 120ebfd34a9e4d73a215cb9206013728 (table default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]) successfully deleted
I20240507 08:11:51.583853 5837 raft_consensus.cc:2229] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.584007 5837 raft_consensus.cc:2258] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.584539 5837 ts_tablet_manager.cc:1875] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.585233 5901 raft_consensus.cc:482] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.585476 5901 raft_consensus.cc:504] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.586542 5901 leader_election.cc:288] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.586844 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ede5b03251463d8608d15f037e4484" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:51.587074 5720 raft_consensus.cc:2454] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:51.587426 5377 leader_election.cc:302] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.587769 5901 raft_consensus.cc:2781] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.587916 5901 raft_consensus.cc:482] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.588073 5901 raft_consensus.cc:3037] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.590811 5901 raft_consensus.cc:504] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.592121 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ede5b03251463d8608d15f037e4484" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:51.592329 5720 raft_consensus.cc:3037] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.594359 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet f29568ebe4774662a8b49b7829cf30e7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.594457 5837 ts_tablet_manager.cc:1888] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.594611 5837 log.cc:1196] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c68b6c08f00d46779a31f4e9eed99f9f
I20240507 08:11:51.594987 5837 ts_tablet_manager.cc:1909] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.595021 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.595619 5837 tablet_replica.cc:330] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.595819 5837 raft_consensus.cc:2229] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:51.595873 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c68b6c08f00d46779a31f4e9eed99f9f (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:51.596151 5720 raft_consensus.cc:2454] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:51.596442 5837 pending_rounds.cc:62] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:51.596763 5837 pending_rounds.cc:68] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924716598476800 op_type: NO_OP noop_request { }
I20240507 08:11:51.596917 5377 leader_election.cc:302] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:51.597007 5837 raft_consensus.cc:2866] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:51.597122 5837 raft_consensus.cc:2258] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.597193 5902 raft_consensus.cc:2781] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.598075 5837 ts_tablet_manager.cc:1875] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.600768 5585 raft_consensus.cc:2454] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:51.601271 5654 leader_election.cc:302] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters:
W20240507 08:11:51.601471 5654 raft_consensus.cc:2670] T c68b6c08f00d46779a31f4e9eed99f9f P 1cbb887175f345ada2433adaec906fd4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:51.601660 5902 raft_consensus.cc:686] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:51.602536 5902 consensus_queue.cc:237] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.604585 5585 raft_consensus.cc:3037] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.605185 5837 ts_tablet_manager.cc:1888] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:51.605370 5837 log.cc:1196] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/19efea8905cf4e83a523c278b2f16dc7
I20240507 08:11:51.605721 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a1158486249b4ac59cd8e9db4042a5d2" } modify_external_catalogs: true
I20240507 08:11:51.605844 5837 ts_tablet_manager.cc:1909] T 19efea8905cf4e83a523c278b2f16dc7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.605998 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a1158486249b4ac59cd8e9db4042a5d2" } modify_external_catalogs: true
I20240507 08:11:51.606928 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 19efea8905cf4e83a523c278b2f16dc7 (table default.Table_237e57be541d40359403f08fcc681dd8 [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:51.607307 5837 tablet_replica.cc:330] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.607551 5837 raft_consensus.cc:2229] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.607664 5843 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.607795 5837 raft_consensus.cc:2258] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.607837 5843 ts_tablet_manager.cc:1367] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20240507 08:11:51.608384 5837 ts_tablet_manager.cc:1875] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.608490 5891 consensus_queue.cc:1035] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:51.608793 5843 raft_consensus.cc:348] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.609045 5843 raft_consensus.cc:374] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.609169 5843 raft_consensus.cc:729] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.609467 5843 consensus_queue.cc:260] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.609776 5901 leader_election.cc:288] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:51.610268 5843 ts_tablet_manager.cc:1398] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:51.613029 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_80a52d858eaf480cab22d552d7c8eff5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.613746 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80a52d858eaf480cab22d552d7c8eff5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.614234 5447 raft_consensus.cc:1261] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Refusing update from remote peer fe673739d8084e7da5d7edd722e857d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:51.614719 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.615051 5446 raft_consensus.cc:2379] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe673739d8084e7da5d7edd722e857d8 in current term 1: Already voted for candidate 1cbb887175f345ada2433adaec906fd4 in this term.
I20240507 08:11:51.615303 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.615572 5445 raft_consensus.cc:2454] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.615813 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400d2a7b0e224e46a5a580ca22e96eab" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.616070 5444 raft_consensus.cc:2454] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.616286 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c68b6c08f00d46779a31f4e9eed99f9f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:51.616472 5443 raft_consensus.cc:3037] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.618432 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 34e4e33f52df432b8d043171d7232c2a (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]). Expected version 1 got 0
I20240507 08:11:51.620424 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_edd5577d442c45a3b978ed71ad918a84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.620554 5693 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (DEFAULT_TABLE table=default.Table_e5a3509f77064f24a47bc7d69395d0c7 [id=b3fb019c81734b0f8108f607f4b15972]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.620947 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea74702c4b4514a31f0fad7e7a2f5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:51.620960 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edd5577d442c45a3b978ed71ad918a84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.621634 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e" } modify_external_catalogs: true
I20240507 08:11:51.621819 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e" } modify_external_catalogs: true
I20240507 08:11:51.622951 5420 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (DEFAULT_TABLE table=default.Table_e5a3509f77064f24a47bc7d69395d0c7 [id=b3fb019c81734b0f8108f607f4b15972]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.623318 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea74702c4b4514a31f0fad7e7a2f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.624801 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5d566695d94c708e90ee30254084a1 (DEFAULT_TABLE table=default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.625144 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5d566695d94c708e90ee30254084a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.625866 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5d566695d94c708e90ee30254084a1 (DEFAULT_TABLE table=default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.626384 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5d566695d94c708e90ee30254084a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.626801 5840 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.626981 5840 ts_tablet_manager.cc:1367] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.014s
I20240507 08:11:51.627668 5840 raft_consensus.cc:348] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.627864 5840 raft_consensus.cc:374] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.627943 5840 raft_consensus.cc:729] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.628175 5840 consensus_queue.cc:260] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.628803 5840 ts_tablet_manager.cc:1398] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:51.629235 5656 leader_election.cc:302] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.630448 5891 raft_consensus.cc:2781] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.630568 5891 raft_consensus.cc:482] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.630709 5891 raft_consensus.cc:3037] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.633304 5891 raft_consensus.cc:504] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.637892 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.638145 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.638312 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.638703 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.639344 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.639575 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.639736 5443 raft_consensus.cc:2454] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:51.639899 5840 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.639962 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 344bf4ab39f74accb383d695578de604
I20240507 08:11:51.641546 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.641561 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.644559 5837 ts_tablet_manager.cc:1888] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.645143 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fcab59c9e0064c3487c28efc45b90928" } modify_external_catalogs: true
I20240507 08:11:51.645210 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0c7582240f584dd8a2969c426b8aeb25" } new_table_name: "default.Table_60d0f9c31e13466793596c325f991a88" modify_external_catalogs: true
I20240507 08:11:51.645354 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fcab59c9e0064c3487c28efc45b90928" } modify_external_catalogs: true
I20240507 08:11:51.646142 5837 log.cc:1196] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/71bbe28a013344bcb6529247d7139ab9
I20240507 08:11:51.647982 5837 ts_tablet_manager.cc:1909] T 71bbe28a013344bcb6529247d7139ab9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:51.648227 5840 tablet_bootstrap.cc:654] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.649039 5837 tablet_replica.cc:330] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:51.649111 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 71bbe28a013344bcb6529247d7139ab9 (table default.Table_1d5ade8f490d40b4a0e5cd4cdc996a66 [id=d6fcb0621d0a4661bd83020163e56886]) successfully deleted
I20240507 08:11:51.649446 5837 raft_consensus.cc:2229] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.651268 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.652897 5837 raft_consensus.cc:2258] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.653172 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.653250 5841 tablet_replica.cc:330] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:51.653546 5841 raft_consensus.cc:2229] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:51.654016 5841 pending_rounds.cc:62] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:51.654778 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cc2b1c5d509b43afa9523de30d68a4d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.654640 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400d2a7b0e224e46a5a580ca22e96eab" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:51.655407 5443 raft_consensus.cc:3037] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:51.655436 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc2b1c5d509b43afa9523de30d68a4d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.657199 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d9ff5e1b6a9a476c8295a0d0ee85aecc
I20240507 08:11:51.657920 5843 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.652657 5848 consensus_queue.cc:1035] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20240507 08:11:51.659161 5443 raft_consensus.cc:2454] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:51.659518 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.660086 5841 pending_rounds.cc:68] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924716962357248 op_type: NO_OP noop_request { }
I20240507 08:11:51.660756 5841 raft_consensus.cc:2866] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:51.661049 5841 raft_consensus.cc:2258] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.663483 5839 tablet_bootstrap.cc:654] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.668819 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_06c4bdeddaf4480f82cec8f330889325" } modify_external_catalogs: true
I20240507 08:11:51.669188 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_06c4bdeddaf4480f82cec8f330889325" } modify_external_catalogs: true
I20240507 08:11:51.670558 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b91d41a813624523a849922b73d8bd03" } new_table_name: "default.Table_7ca5eb8b7d094b3bb020f4fd128257fa" modify_external_catalogs: true
I20240507 08:11:51.670866 5420 tablet_service.cc:1467] Processing CreateTablet for tablet f922a7519084409db98693ee8797592b (DEFAULT_TABLE table=default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.671255 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f922a7519084409db98693ee8797592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.673036 5843 tablet_bootstrap.cc:654] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.676059 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708d4bee30cc45bd9b86c56beb2c3ece" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.676820 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d58aa20be2d34d0e9ec80656b522d017" } new_table_name: "default.Table_a2f27f59d07b4b23a4157ff49c53b62c" modify_external_catalogs: true
I20240507 08:11:51.677718 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5" } new_table_name: "default.Table_e85104a6436841ad8d9510f2e54233c3" modify_external_catalogs: true
I20240507 08:11:51.678205 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e97ccca6ca324143a7e713d916068114" } new_table_name: "default.Table_70320eb4c56d4c38b5f30235746e7cb2" modify_external_catalogs: true
I20240507 08:11:51.681707 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:51.690362 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60bc7bc06e494adcb6a8641b50b90e6c consensus state has no leader
I20240507 08:11:51.691074 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet f29568ebe4774662a8b49b7829cf30e7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.691200 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f922a7519084409db98693ee8797592b
I20240507 08:11:51.693054 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.692756 5423 tablet_service.cc:1467] Processing CreateTablet for tablet c1c6b488b9014841899597d858bda5e2 (DEFAULT_TABLE table=default.Table_f956db8e0c684a60a84296ba0504178d [id=0a1bc06a351549d5aea075ad8d026476]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.693519 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c6b488b9014841899597d858bda5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.690289 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "400d2a7b0e224e46a5a580ca22e96eab" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.700487 5838 ts_tablet_manager.cc:1888] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.700608 5838 log.cc:1196] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/19efea8905cf4e83a523c278b2f16dc7
I20240507 08:11:51.702253 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.703706 5695 tablet_service.cc:1467] Processing CreateTablet for tablet f922a7519084409db98693ee8797592b (DEFAULT_TABLE table=default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.703843 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.704165 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f922a7519084409db98693ee8797592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.704542 5693 tablet_service.cc:1467] Processing CreateTablet for tablet c1c6b488b9014841899597d858bda5e2 (DEFAULT_TABLE table=default.Table_f956db8e0c684a60a84296ba0504178d [id=0a1bc06a351549d5aea075ad8d026476]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.704815 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:51.704885 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c6b488b9014841899597d858bda5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.705655 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ebc6cc72bb7e4721a8ab559102273db5
I20240507 08:11:51.706848 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_390a0d927d2c43f997940b8fed566a5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.707423 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_390a0d927d2c43f997940b8fed566a5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.707477 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_84d691a98c0d402fb6532f0e49f61488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.707945 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84d691a98c0d402fb6532f0e49f61488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.709955 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.710072 5891 leader_election.cc:288] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.703753 5656 leader_election.cc:302] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:51.710294 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 50431.4us
W20240507 08:11:51.710480 5654 leader_election.cc:341] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.711154 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:51.711442 5696 tablet_service.cc:1467] Processing CreateTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (DEFAULT_TABLE table=default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.711536 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.711833 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.711781 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.711938 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4867e1cca1634b4ab6e36f9ed3204e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.712133 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4867e1cca1634b4ab6e36f9ed3204e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.711540 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 270a1fd15925416483c211021549315c (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.713313 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270a1fd15925416483c211021549315c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.713723 5840 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:51.713899 5840 ts_tablet_manager.cc:1367] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.074s user 0.014s sys 0.017s
I20240507 08:11:51.714740 5840 raft_consensus.cc:348] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.715015 5840 raft_consensus.cc:374] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.715101 5840 raft_consensus.cc:729] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.715881 5840 consensus_queue.cc:260] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.716426 5840 ts_tablet_manager.cc:1398] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:11:51.717730 5851 raft_consensus.cc:2781] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:51.717732 5419 tablet_service.cc:1467] Processing CreateTablet for tablet f49dd4772f8e46478638b29c009c83ea (DEFAULT_TABLE table=default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.718056 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49dd4772f8e46478638b29c009c83ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.722754 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc (DEFAULT_TABLE table=default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.722920 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 (DEFAULT_TABLE table=default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.723137 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf0e2936324a9fbb2426a8026886fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.723263 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c (DEFAULT_TABLE table=default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.723496 5840 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:51.723592 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bc7bc06e494adcb6a8641b50b90e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.725050 5840 tablet_bootstrap.cc:654] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.725134 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6670e99e4a4442801d42950eb7ab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.731334 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec (DEFAULT_TABLE table=default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.731711 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf34089e5ec419f97c0da9ec42d7eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.736721 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 270a1fd15925416483c211021549315c (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.737453 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0b02bc193ec84c7abcc0754d885cfb89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.737902 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270a1fd15925416483c211021549315c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:51.738062 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b02bc193ec84c7abcc0754d885cfb89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.737902 5699 tablet_service.cc:1467] Processing CreateTablet for tablet f49dd4772f8e46478638b29c009c83ea (DEFAULT_TABLE table=default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.700296 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "708d4bee30cc45bd9b86c56beb2c3ece" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.743710 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49dd4772f8e46478638b29c009c83ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.743919 5907 raft_consensus.cc:482] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.747313 5843 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.747502 5843 ts_tablet_manager.cc:1367] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.000s
I20240507 08:11:51.748337 5843 raft_consensus.cc:348] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.748566 5843 raft_consensus.cc:374] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:51.748637 5843 raft_consensus.cc:729] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:51.749004 5263 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 87252a9cf30c45fdb43854b13bc079da
I20240507 08:11:51.748980 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5cfe3c25a9a84eab9091502955453951 (table default.Table_a1e20af4278a4c35ab581d660cc9ec3e [id=70026f101a8d4a1db79650e8e1f66c57]) successfully deleted
I20240507 08:11:51.748903 5843 consensus_queue.cc:260] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:51.749262 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 34ffe98d424a4eb5a08be067f96a50f9 because the tablet was not found. No further retry: Not found: Tablet not found: 34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:51.749483 5843 ts_tablet_manager.cc:1398] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240507 08:11:51.749490 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ea17889619fa420fbeacf1b41594c598 because the tablet was not found. No further retry: Not found: Tablet not found: ea17889619fa420fbeacf1b41594c598
W20240507 08:11:51.750666 5378 leader_election.cc:341] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.753628 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ede5b03251463d8608d15f037e4484" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:51.754069 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87ede5b03251463d8608d15f037e4484" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:51.757050 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 91c6cb7920df47aa97be6555c587798a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c6cb7920df47aa97be6555c587798a already in progress: creating tablet
W20240507 08:11:51.762615 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4899c088531480ea15364af94595ee5 consensus state has no leader
W20240507 08:11:51.763372 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ba24c078af9472da06793c58f147290 consensus state has no leader
I20240507 08:11:51.760150 5838 ts_tablet_manager.cc:1909] T 19efea8905cf4e83a523c278b2f16dc7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:51.765673 5838 tablet_replica.cc:330] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.765884 5838 raft_consensus.cc:2229] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:51.767393 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 812dd1ebd42f4e929838a0535c22ad69 consensus state has no leader
I20240507 08:11:51.767827 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_31c7cf5701db40cebfa0c46ebaae046c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.768370 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31c7cf5701db40cebfa0c46ebaae046c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:51.770011 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7eaf0e2936324a9fbb2426a8026886fc consensus state has no leader
I20240507 08:11:51.783907 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:51.787660 5843 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:51.787704 5423 tablet_service.cc:1467] Processing CreateTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (DEFAULT_TABLE table=default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.788164 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.788573 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 191adef34efa4c3999d4642a80177294 (DEFAULT_TABLE table=default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.788964 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191adef34efa4c3999d4642a80177294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.790182 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0fb75933874a948281bf573a1723a1 (DEFAULT_TABLE table=default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.790553 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0fb75933874a948281bf573a1723a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.791100 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet e1cf2df0d3bd46268e22713fcc135a74 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.791654 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b8268009436d4ce088ecbc9aae6f3fdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.792191 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8268009436d4ce088ecbc9aae6f3fdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.793640 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ce67a10a1e3f48a8b7225aad61e776e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.793767 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ffaf1781e4b108f305346476905be (DEFAULT_TABLE table=default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:51.794111 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce67a10a1e3f48a8b7225aad61e776e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.794780 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ffaf1781e4b108f305346476905be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.795848 5843 tablet_bootstrap.cc:654] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:51.797823 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f (DEFAULT_TABLE table=default.Table_ab41152f87d9489585412dcc24dd2098 [id=2e71cf49719f4d9fb07c289e1428080c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.798336 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed8bac124cb4de983f0fdbaf728975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.804984 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:51.805667 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4a583347a22943338c2cad0211f8f17e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.806124 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a583347a22943338c2cad0211f8f17e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.807634 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 4306751d16444ec5948c2ba5831063f4 (DEFAULT_TABLE table=default.Table_b67c0c470e744311924705a8842f1926 [id=a005b1852f254fd183b3323e3d5ac6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.808004 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4306751d16444ec5948c2ba5831063f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.812994 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.815721 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53409fe13e4a9585dbba648a25cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.816589 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 88a2cfb5f046434c8340234a447a7e65 (DEFAULT_TABLE table=default.Table_edd5577d442c45a3b978ed71ad918a84 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.817942 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a2cfb5f046434c8340234a447a7e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.821450 5420 tablet_service.cc:1467] Processing CreateTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (DEFAULT_TABLE table=default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.821833 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.825860 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 8615f89e816e48fc8a058e5dc466918c (DEFAULT_TABLE table=default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.827023 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 364903504aa14d08b050a0cf83b82c71 (DEFAULT_TABLE table=default.Table_d9f6ed618e184cc6aece7c2d0dca9df8 [id=95b44cbcf26f489c8f4d8e07cc615776]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.827733 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364903504aa14d08b050a0cf83b82c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.833513 5422 tablet_service.cc:1467] Processing CreateTablet for tablet bde62374dc7f4a67a071b6b658e501c4 (DEFAULT_TABLE table=default.Table_923d5aad72744e84a98819df252a2e2c [id=d0a57f2571ca478084824a44922ac99b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:51.834074 5520 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 135528us
I20240507 08:11:51.834275 5417 tablet_service.cc:1467] Processing CreateTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (DEFAULT_TABLE table=default.Table_914ee05f51f14d84bfc1418f5a9cfb27 [id=70d5dddc40a3453bb64b8019c2a18046]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.835373 5950 mvcc.cc:205] Tried to move back new op lower bound from 7024924719072206848 to 7024924716715016192. Current Snapshot: MvccSnapshot[applied={T|T < 7024924719072206848}]
I20240507 08:11:51.835522 5838 raft_consensus.cc:2258] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.835719 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:51.836114 5838 ts_tablet_manager.cc:1875] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.836913 5426 tablet_service.cc:1467] Processing CreateTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 (DEFAULT_TABLE table=default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.837419 5423 tablet_service.cc:1467] Processing CreateTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (DEFAULT_TABLE table=default.Table_7004d9f9040b4d57abb409a848b45e48 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.841114 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet c68b6c08f00d46779a31f4e9eed99f9f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.841768 5561 tablet_service.cc:1467] Processing CreateTablet for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (DEFAULT_TABLE table=default.Table_e5a3509f77064f24a47bc7d69395d0c7 [id=b3fb019c81734b0f8108f607f4b15972]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.842129 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0ea74702c4b4514a31f0fad7e7a2f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.841492 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet cbf73c218b944679a12d769b1929e535 (table default.Table_9152edd2e429414ea4670ea7d1ae0cc4 [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 3 got 0
I20240507 08:11:51.846483 5838 ts_tablet_manager.cc:1888] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.846630 5838 log.cc:1196] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/81919c8f720f4c72bc2cc457b095859b
I20240507 08:11:51.846992 5838 ts_tablet_manager.cc:1909] T 81919c8f720f4c72bc2cc457b095859b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:51.847720 5838 tablet_replica.cc:330] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.847916 5838 raft_consensus.cc:2229] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.848110 5838 raft_consensus.cc:2258] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.848070 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eff2f49d93764317a595c308278c0732" } modify_external_catalogs: true
I20240507 08:11:51.848261 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eff2f49d93764317a595c308278c0732" } modify_external_catalogs: true
I20240507 08:11:51.848623 5838 ts_tablet_manager.cc:1875] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.851287 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8615f89e816e48fc8a058e5dc466918c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.852528 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ffaf1781e4b108f305346476905be (DEFAULT_TABLE table=default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.853549 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a0ac2efb18004de2977210793a4b8de7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.854209 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0ac2efb18004de2977210793a4b8de7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.856962 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c (DEFAULT_TABLE table=default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.859789 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8d15746afd6042b29e476bb2ecfa6c6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.860999 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f053f92b620148968b86e702410b211d" } modify_external_catalogs: true
I20240507 08:11:51.861191 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f053f92b620148968b86e702410b211d" } modify_external_catalogs: true
I20240507 08:11:51.861347 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc (DEFAULT_TABLE table=default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.860009 5907 raft_consensus.cc:504] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:51.862784 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.863106 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:51.863348 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d15746afd6042b29e476bb2ecfa6c6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.863396 5891 consensus_queue.cc:1035] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20240507 08:11:51.864558 5837 ts_tablet_manager.cc:1875] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.866149 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 4306751d16444ec5948c2ba5831063f4 (DEFAULT_TABLE table=default.Table_b67c0c470e744311924705a8842f1926 [id=a005b1852f254fd183b3323e3d5ac6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.868572 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:51.869212 5696 tablet_service.cc:1467] Processing CreateTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 (DEFAULT_TABLE table=default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.864667 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf73c218b944679a12d769b1929e535" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:51.871308 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ffaf1781e4b108f305346476905be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.871481 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:51.871666 5907 leader_election.cc:288] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:51.872288 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0fb75933874a948281bf573a1723a1 (DEFAULT_TABLE table=default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:51.873378 5654 leader_election.cc:341] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:51.873615 5720 raft_consensus.cc:1261] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Refusing update from remote peer fe673739d8084e7da5d7edd722e857d8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:51.874009 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 (DEFAULT_TABLE table=default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.878839 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5228590c8e7d4c34b3acab9eb16665d3
I20240507 08:11:51.873155 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 191adef34efa4c3999d4642a80177294 (DEFAULT_TABLE table=default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.873903 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f (DEFAULT_TABLE table=default.Table_ab41152f87d9489585412dcc24dd2098 [id=2e71cf49719f4d9fb07c289e1428080c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.874382 5686 tablet_service.cc:1467] Processing CreateTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (DEFAULT_TABLE table=default.Table_914ee05f51f14d84bfc1418f5a9cfb27 [id=70d5dddc40a3453bb64b8019c2a18046]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.874481 5685 tablet_service.cc:1467] Processing CreateTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (DEFAULT_TABLE table=default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.878500 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 88a2cfb5f046434c8340234a447a7e65 (DEFAULT_TABLE table=default.Table_edd5577d442c45a3b978ed71ad918a84 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.878726 5682 tablet_service.cc:1467] Processing CreateTablet for tablet bde62374dc7f4a67a071b6b658e501c4 (DEFAULT_TABLE table=default.Table_923d5aad72744e84a98819df252a2e2c [id=d0a57f2571ca478084824a44922ac99b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.880290 5838 ts_tablet_manager.cc:1888] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.878955 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 364903504aa14d08b050a0cf83b82c71 (DEFAULT_TABLE table=default.Table_d9f6ed618e184cc6aece7c2d0dca9df8 [id=95b44cbcf26f489c8f4d8e07cc615776]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.880816 5838 log.cc:1196] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5cb62017acbd4472ac5b143eafb36ae9
I20240507 08:11:51.879164 5687 tablet_service.cc:1467] Processing CreateTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (DEFAULT_TABLE table=default.Table_7004d9f9040b4d57abb409a848b45e48 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.882699 5838 ts_tablet_manager.cc:1909] T 5cb62017acbd4472ac5b143eafb36ae9 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:51.883530 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec (DEFAULT_TABLE table=default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.884588 5838 tablet_replica.cc:330] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:51.884999 5838 raft_consensus.cc:2229] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:51.885185 5838 raft_consensus.cc:2258] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:51.886552 5838 ts_tablet_manager.cc:1875] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.886616 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4306751d16444ec5948c2ba5831063f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.889694 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf0e2936324a9fbb2426a8026886fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.891901 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9152edd2e429414ea4670ea7d1ae0cc4" } new_table_name: "default.Table_4dbb2a240d8349e8b24875b8f9e99eab" modify_external_catalogs: true
I20240507 08:11:51.897133 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_60d0f9c31e13466793596c325f991a88" } modify_external_catalogs: true
I20240507 08:11:51.897328 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_60d0f9c31e13466793596c325f991a88" } modify_external_catalogs: true
I20240507 08:11:51.898811 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde460fdcf254af6a3cdaaac22ac2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.901387 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5d566695d94c708e90ee30254084a1 (DEFAULT_TABLE table=default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.903385 5902 raft_consensus.cc:482] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.903645 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5d566695d94c708e90ee30254084a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.909714 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.911224 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:51.911375 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_169e8150dda24f87ac304a6c9031909d" } modify_external_catalogs: true
I20240507 08:11:51.911556 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_169e8150dda24f87ac304a6c9031909d" } modify_external_catalogs: true
I20240507 08:11:51.912140 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c93fda7fb0754945b3d3764e5698344e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:51.912361 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:51.912678 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c93fda7fb0754945b3d3764e5698344e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.914412 5838 ts_tablet_manager.cc:1888] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:51.914448 5556 tablet_service.cc:1467] Processing CreateTablet for tablet f922a7519084409db98693ee8797592b (DEFAULT_TABLE table=default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.914834 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f922a7519084409db98693ee8797592b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.916502 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 60bc7bc06e494adcb6a8641b50b90e6c
I20240507 08:11:51.918229 5562 tablet_service.cc:1467] Processing CreateTablet for tablet c1c6b488b9014841899597d858bda5e2 (DEFAULT_TABLE table=default.Table_f956db8e0c684a60a84296ba0504178d [id=0a1bc06a351549d5aea075ad8d026476]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.918597 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1c6b488b9014841899597d858bda5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.924829 5839 tablet_bootstrap.cc:492] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:51.925860 5839 ts_tablet_manager.cc:1367] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.360s user 0.003s sys 0.011s
I20240507 08:11:51.929463 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 8615f89e816e48fc8a058e5dc466918c (DEFAULT_TABLE table=default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.931969 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf34089e5ec419f97c0da9ec42d7eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.934587 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet cbf73c218b944679a12d769b1929e535 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.937099 5901 raft_consensus.cc:482] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.938360 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 270a1fd15925416483c211021549315c
I20240507 08:11:51.939358 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet f968f122c39b4fd5b995e8a2848a52d4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.940603 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 87ede5b03251463d8608d15f037e4484 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:51.914855 5838 log.cc:1196] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c68b6c08f00d46779a31f4e9eed99f9f
I20240507 08:11:51.944666 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cbf73c218b944679a12d769b1929e535
I20240507 08:11:51.945163 5263 catalog_manager.cc:6554] ReplaceTablet: tablet cbf73c218b944679a12d769b1929e535 deleted and replaced by tablet 169626f9a0bc4c6f9d8be11c0a802816
I20240507 08:11:51.950415 5841 ts_tablet_manager.cc:1875] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:51.951889 5902 raft_consensus.cc:504] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:51.951803 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 34bebec088a34ebbb06d2e707d54da4c (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.956238 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_31c7cf5701db40cebfa0c46ebaae046c" } modify_external_catalogs: true
I20240507 08:11:51.956486 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_31c7cf5701db40cebfa0c46ebaae046c" } modify_external_catalogs: true
I20240507 08:11:51.962342 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde62374dc7f4a67a071b6b658e501c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.965497 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_923d5aad72744e84a98819df252a2e2c" } new_table_name: "default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3" modify_external_catalogs: true
I20240507 08:11:51.966756 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_12be08cdc73d4d488affa98e5f2d2653"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.967273 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12be08cdc73d4d488affa98e5f2d2653 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.968722 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bbe018a0c26f42e2988b531cbf4df1d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.969296 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbe018a0c26f42e2988b531cbf4df1d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.970425 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_522ea36e321d413c977697e907f9e118"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:51.970870 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_522ea36e321d413c977697e907f9e118 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:51.974004 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7ca5eb8b7d094b3bb020f4fd128257fa" } new_table_name: "default.Table_8ad438587c5c48c78d151533b16a55e3" modify_external_catalogs: true
I20240507 08:11:51.973502 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0b02bc193ec84c7abcc0754d885cfb89" } new_table_name: "default.Table_c8286bc81dc24b5e8159136211d18799" modify_external_catalogs: true
I20240507 08:11:51.978796 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:51.979096 5445 raft_consensus.cc:2454] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:51.984335 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:51.986157 5843 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:51.986337 5843 ts_tablet_manager.cc:1367] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.232s user 0.003s sys 0.004s
I20240507 08:11:51.988960 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 5554ebc621c24d459668f2cefc20b641 (DEFAULT_TABLE table=default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:51.989519 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7c1aaf55d74c979ec1fc0a57f15f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.992617 5656 leader_election.cc:302] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:51.993151 5987 raft_consensus.cc:2781] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:51.993258 5987 raft_consensus.cc:482] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:51.993391 5987 raft_consensus.cc:3037] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:51.999235 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ab41152f87d9489585412dcc24dd2098" } new_table_name: "default.Table_b430c7d7295e4ca6b22aaf7c02ac328f" modify_external_catalogs: true
I20240507 08:11:52.000543 5907 raft_consensus.cc:482] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.000686 5907 raft_consensus.cc:504] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.001014 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f68ddcc012de4bd9ba219332fbfa8760"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.001359 5907 leader_election.cc:288] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:52.001539 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f68ddcc012de4bd9ba219332fbfa8760 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.002045 5840 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:52.002775 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.005759 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bfe2378493e64efb93444e140f410dfe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.006232 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfe2378493e64efb93444e140f410dfe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.008131 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5554ebc621c24d459668f2cefc20b641
I20240507 08:11:52.009057 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ca27a6c33fe74eff87c2e753756029f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.009691 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca27a6c33fe74eff87c2e753756029f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.017764 5901 raft_consensus.cc:504] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.018460 5901 leader_election.cc:288] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.018630 5901 raft_consensus.cc:482] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.018754 5901 raft_consensus.cc:504] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.019364 5901 leader_election.cc:288] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:52.019452 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bde62374dc7f4a67a071b6b658e501c4 consensus state has no leader
I20240507 08:11:52.019959 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a58b7d46a51f41049815a957ddac3f55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.020483 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a58b7d46a51f41049815a957ddac3f55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.021960 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4899c088531480ea15364af94595ee5 consensus state has no leader
I20240507 08:11:52.022361 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed8bac124cb4de983f0fdbaf728975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:51.996174 5987 raft_consensus.cc:504] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.027143 5840 ts_tablet_manager.cc:1367] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.279s user 0.012s sys 0.025s
I20240507 08:11:52.029002 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8615f89e816e48fc8a058e5dc466918c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.029129 5851 raft_consensus.cc:686] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:52.029393 5851 consensus_queue.cc:237] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.031046 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.031566 5987 leader_election.cc:288] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.032492 5978 raft_consensus.cc:482] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:51.987497 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 072702ef2a6d43248132da6aa3a2d493 (DEFAULT_TABLE table=default.Table_ce67a10a1e3f48a8b7225aad61e776e0 [id=3d99e1ae7dc34ac49843ce5f159f891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.039225 5992 tablet.cc:1722] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:52.039603 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 077ccc3946b34008bea7b80d8aee2662 (DEFAULT_TABLE table=default.Table_84d691a98c0d402fb6532f0e49f61488 [id=34d5ccf8d18945608504d42ba58ad664]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.039661 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:52.039860 5445 raft_consensus.cc:3037] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.039950 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.039994 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554ebc621c24d459668f2cefc20b641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.040095 5843 raft_consensus.cc:348] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.041636 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 072702ef2a6d43248132da6aa3a2d493 (DEFAULT_TABLE table=default.Table_ce67a10a1e3f48a8b7225aad61e776e0 [id=3d99e1ae7dc34ac49843ce5f159f891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.037191 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f6f62c56cb4c29bf05914ca461a559" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:52.043288 5720 raft_consensus.cc:2454] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:52.047025 5837 ts_tablet_manager.cc:1888] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:52.047910 5837 log.cc:1196] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/717460af19164c058f42c7a03095bb30
I20240507 08:11:52.041287 5902 leader_election.cc:288] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.048836 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde62374dc7f4a67a071b6b658e501c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.048905 5843 raft_consensus.cc:374] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.054140 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:52.055426 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bebec088a34ebbb06d2e707d54da4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.052999 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3e8b4d5d594f378451009c313f6bfa (DEFAULT_TABLE table=default.Table_4a583347a22943338c2cad0211f8f17e [id=06457cc0f83a470a95273688e948d21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.056217 5837 ts_tablet_manager.cc:1909] T 717460af19164c058f42c7a03095bb30 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:52.047657 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 34bebec088a34ebbb06d2e707d54da4c (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.055058 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 (DEFAULT_TABLE table=default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.059651 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (DEFAULT_TABLE table=default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.061519 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.061501 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc (DEFAULT_TABLE table=default.Table_0b02bc193ec84c7abcc0754d885cfb89 [id=0e277dd692e842b28638b8675da40ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.061830 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072702ef2a6d43248132da6aa3a2d493. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.062125 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed8bac124cb4de983f0fdbaf728975f consensus state has no leader
I20240507 08:11:52.065680 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.066100 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 5554ebc621c24d459668f2cefc20b641 (DEFAULT_TABLE table=default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.066308 5843 raft_consensus.cc:729] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.066799 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364903504aa14d08b050a0cf83b82c71. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.067376 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 907b5cb0e7a54903abe0b0d0e8d617dc consensus state has no leader
I20240507 08:11:52.040305 5978 raft_consensus.cc:504] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.069289 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.069944 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 077ccc3946b34008bea7b80d8aee2662 (DEFAULT_TABLE table=default.Table_84d691a98c0d402fb6532f0e49f61488 [id=34d5ccf8d18945608504d42ba58ad664]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.071408 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a2cfb5f046434c8340234a447a7e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.074321 5445 raft_consensus.cc:2454] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:52.074350 5424 tablet_service.cc:1467] Processing CreateTablet for tablet d537bed29a2f47058255d2530b5c0d23 (DEFAULT_TABLE table=default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.076064 5656 leader_election.cc:302] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.074826 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.077168 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0fb75933874a948281bf573a1723a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.077226 5443 raft_consensus.cc:2454] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:52.077652 5978 leader_election.cc:288] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.078907 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7c1aaf55d74c979ec1fc0a57f15f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.079573 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6670e99e4a4442801d42950eb7ab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.080469 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bc7bc06e494adcb6a8641b50b90e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.078501 5843 consensus_queue.cc:260] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.083508 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077ccc3946b34008bea7b80d8aee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.084803 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d455ec93c6a1445a8948ee6973d6e4c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.085403 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d455ec93c6a1445a8948ee6973d6e4c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.086262 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d537bed29a2f47058255d2530b5c0d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.086503 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b43daab6b4deeac94fc6f1ff34a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.088624 5987 raft_consensus.cc:2781] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:52.089001 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.089154 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_390a0d927d2c43f997940b8fed566a5a" } modify_external_catalogs: true
I20240507 08:11:52.089344 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_390a0d927d2c43f997940b8fed566a5a" } modify_external_catalogs: true
I20240507 08:11:52.078697 5410 tablet_service.cc:1467] Processing CreateTablet for tablet b2de528d9fc046f183f70388cffab6c8 (DEFAULT_TABLE table=default.Table_8d15746afd6042b29e476bb2ecfa6c6e [id=317e03893acd43ef95533e458ed4346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.090704 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30365e8917ec45c0afafa640e07f03dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.091578 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30365e8917ec45c0afafa640e07f03dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.092864 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3e8b4d5d594f378451009c313f6bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.095486 5656 leader_election.cc:302] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.096762 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9a56a95074e67bb33ceee1af48a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.096935 5843 ts_tablet_manager.cc:1398] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.098s user 0.002s sys 0.001s
I20240507 08:11:52.102444 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf73c218b944679a12d769b1929e535 with delete_type TABLET_DATA_DELETED (replaced by tablet 169626f9a0bc4c6f9d8be11c0a802816) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.102638 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:52.103994 5838 ts_tablet_manager.cc:1909] T c68b6c08f00d46779a31f4e9eed99f9f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:52.104025 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.104697 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2de528d9fc046f183f70388cffab6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.105265 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.107206 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907b5cb0e7a54903abe0b0d0e8d617dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.106299 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 916e33e40ef44d06a359b82d8fe48157 (DEFAULT_TABLE table=default.Table_c93fda7fb0754945b3d3764e5698344e [id=14589abdd357409cbe9837a33208e00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.108242 5843 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:52.112948 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_522ea36e321d413c977697e907f9e118" } modify_external_catalogs: true
I20240507 08:11:52.113168 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_522ea36e321d413c977697e907f9e118" } modify_external_catalogs: true
I20240507 08:11:52.114008 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 717460af19164c058f42c7a03095bb30 (table default.Table_8051479eeaf74dd8beccf13762a64249 [id=5d190d55ad4748cca262a1a33e12b68a]) successfully deleted
I20240507 08:11:52.114413 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e33e40ef44d06a359b82d8fe48157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.116204 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:52.117942 5563 tablet_service.cc:1467] Processing CreateTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (DEFAULT_TABLE table=default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.121778 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 169626f9a0bc4c6f9d8be11c0a802816 (DEFAULT_TABLE table=default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.122678 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.121853 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b430c7d7295e4ca6b22aaf7c02ac328f" } new_table_name: "default.Table_71f59cb7918c42d29419261d5f361830" modify_external_catalogs: true
I20240507 08:11:52.125169 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169626f9a0bc4c6f9d8be11c0a802816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.125885 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbe018a0c26f42e2988b531cbf4df1d2" } modify_external_catalogs: true
I20240507 08:11:52.126098 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbe018a0c26f42e2988b531cbf4df1d2" } modify_external_catalogs: true
I20240507 08:11:52.128782 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.105001 5991 tablet.cc:1722] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:52.127655 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:52.134042 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_889601b1d99641d8b8420888cf849ec1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.134749 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_889601b1d99641d8b8420888cf849ec1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.136504 5377 leader_election.cc:302] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.139513 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a (DEFAULT_TABLE table=default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.140146 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4867e1cca1634b4ab6e36f9ed3204e0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.131047 5839 raft_consensus.cc:348] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:52.141471 5837 tablet_replica.cc:330] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:52.141976 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 270a1fd15925416483c211021549315c (DEFAULT_TABLE table=default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.142385 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 270a1fd15925416483c211021549315c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.147745 5839 raft_consensus.cc:374] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.148726 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a477a636c30a4d289f6c00d2975f1e5c" } modify_external_catalogs: true
I20240507 08:11:52.149109 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a477a636c30a4d289f6c00d2975f1e5c" } modify_external_catalogs: true
I20240507 08:11:52.151862 5558 tablet_service.cc:1467] Processing CreateTablet for tablet f49dd4772f8e46478638b29c009c83ea (DEFAULT_TABLE table=default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.155139 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed8bac124cb4de983f0fdbaf728975f consensus state has no leader
I20240507 08:11:52.155197 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f49dd4772f8e46478638b29c009c83ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.167795 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5089db17ca904a04878d83209001c54e
I20240507 08:11:52.170750 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 400d2a7b0e224e46a5a580ca22e96eab from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.170751 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 09adb78ad08a4e759eb9344cbabb6a9d
I20240507 08:11:52.171792 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c (DEFAULT_TABLE table=default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.172202 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60bc7bc06e494adcb6a8641b50b90e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.173496 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5f2b43daab6b4deeac94fc6f1ff34a76
I20240507 08:11:52.173314 5839 raft_consensus.cc:729] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.177551 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 5228590c8e7d4c34b3acab9eb16665d3 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.178928 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:11:52.180648 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet f29568ebe4774662a8b49b7829cf30e7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.181370 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 44f6f62c56cb4c29bf05914ca461a559 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.181807 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 (DEFAULT_TABLE table=default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.181878 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 34e4e33f52df432b8d043171d7232c2a from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.182220 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f6670e99e4a4442801d42950eb7ab79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.179488 5839 consensus_queue.cc:260] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:52.183702 5320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5089db17ca904a04878d83209001c54e
I20240507 08:11:52.185448 5320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:11:52.190068 5843 tablet_bootstrap.cc:654] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:52.190729 5891 raft_consensus.cc:2781] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:52.190774 5907 raft_consensus.cc:482] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.190881 5891 raft_consensus.cc:482] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:52.191656 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc (DEFAULT_TABLE table=default.Table_9fe8873ddbdb4956b5b6fd9cf74adcc5 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.192075 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7eaf0e2936324a9fbb2426a8026886fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.192366 5837 raft_consensus.cc:2229] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:52.192953 5987 raft_consensus.cc:686] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:11:52.193374 5654 leader_election.cc:341] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.193518 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:52.193703 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 90345.3us
I20240507 08:11:52.193876 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.194969 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 8e3ffaf1781e4b108f305346476905be (DEFAULT_TABLE table=default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.195114 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53409fe13e4a9585dbba648a25cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.195151 5907 raft_consensus.cc:504] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.195459 5891 raft_consensus.cc:3037] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.195504 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e3ffaf1781e4b108f305346476905be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.195991 5837 raft_consensus.cc:2258] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:52.196125 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.196569 5837 ts_tablet_manager.cc:1875] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:52.197126 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077ccc3946b34008bea7b80d8aee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.197748 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde460fdcf254af6a3cdaaac22ac2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.198668 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554ebc621c24d459668f2cefc20b641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.199291 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bebec088a34ebbb06d2e707d54da4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.200109 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191adef34efa4c3999d4642a80177294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.201401 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072702ef2a6d43248132da6aa3a2d493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.183194 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec (DEFAULT_TABLE table=default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.203203 5987 consensus_queue.cc:237] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.204213 5839 ts_tablet_manager.cc:1398] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.269s user 0.001s sys 0.002s
I20240507 08:11:52.205235 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6bf34089e5ec419f97c0da9ec42d7eec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.206429 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.206822 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.213047 5994 raft_consensus.cc:2781] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:52.207878 5840 raft_consensus.cc:348] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.216048 5907 leader_election.cc:288] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.217062 5840 raft_consensus.cc:374] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.217398 5840 raft_consensus.cc:729] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.217983 5840 consensus_queue.cc:260] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.226351 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:52.226625 5444 raft_consensus.cc:2454] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:52.227888 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12be08cdc73d4d488affa98e5f2d2653" } modify_external_catalogs: true
I20240507 08:11:52.228075 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12be08cdc73d4d488affa98e5f2d2653" } modify_external_catalogs: true
I20240507 08:11:52.229936 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 191adef34efa4c3999d4642a80177294 (DEFAULT_TABLE table=default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.230413 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 191adef34efa4c3999d4642a80177294. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.231153 5656 leader_election.cc:302] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.231879 5840 ts_tablet_manager.cc:1398] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.192s user 0.000s sys 0.003s
I20240507 08:11:52.235617 5891 raft_consensus.cc:504] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.237991 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f (DEFAULT_TABLE table=default.Table_ab41152f87d9489585412dcc24dd2098 [id=2e71cf49719f4d9fb07c289e1428080c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.248961 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 34e4e33f52df432b8d043171d7232c2a
I20240507 08:11:52.250003 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ed8bac124cb4de983f0fdbaf728975f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.250682 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 34e4e33f52df432b8d043171d7232c2a deleted and replaced by tablet 98aa4ec7b93943e5bf8f9758b3bb71e0
I20240507 08:11:52.253643 5839 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:52.257421 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0fb75933874a948281bf573a1723a1 (DEFAULT_TABLE table=default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.258285 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 4306751d16444ec5948c2ba5831063f4 (DEFAULT_TABLE table=default.Table_b67c0c470e744311924705a8842f1926 [id=a005b1852f254fd183b3323e3d5ac6a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.261195 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4306751d16444ec5948c2ba5831063f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.262440 5261 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d] alter complete (version 2)
I20240507 08:11:52.263505 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.265659 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0fb75933874a948281bf573a1723a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.265769 5840 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:52.266422 5891 leader_election.cc:288] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.266608 5891 consensus_queue.cc:1035] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:52.268265 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e53409fe13e4a9585dbba648a25cc07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.270044 5563 tablet_service.cc:1467] Processing CreateTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 (DEFAULT_TABLE table=default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.272708 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde460fdcf254af6a3cdaaac22ac2d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.271291 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 8615f89e816e48fc8a058e5dc466918c (DEFAULT_TABLE table=default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.275923 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c93fda7fb0754945b3d3764e5698344e" } new_table_name: "default.Table_992460c354bb4a4c9b51dcf2db8657ae" modify_external_catalogs: true
I20240507 08:11:52.276685 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (DEFAULT_TABLE table=default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.277091 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b43daab6b4deeac94fc6f1ff34a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.277683 5907 raft_consensus.cc:2781] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:52.278399 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5e90ac34a69740dca680fc75e5738dcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.275415 5552 tablet_service.cc:1467] Processing CreateTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (DEFAULT_TABLE table=default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.280547 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e90ac34a69740dca680fc75e5738dcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.282061 5907 raft_consensus.cc:482] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:52.283573 5907 raft_consensus.cc:3037] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.287333 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c33710c6b55744c4ba0843c228abded3
I20240507 08:11:52.299953 5686 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 (DEFAULT_TABLE table=default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.300400 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9a56a95074e67bb33ceee1af48a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.317026 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3e8b4d5d594f378451009c313f6bfa (DEFAULT_TABLE table=default.Table_4a583347a22943338c2cad0211f8f17e [id=06457cc0f83a470a95273688e948d21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.317451 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3e8b4d5d594f378451009c313f6bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.320730 5840 tablet_bootstrap.cc:654] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:52.284657 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 88a2cfb5f046434c8340234a447a7e65 (DEFAULT_TABLE table=default.Table_edd5577d442c45a3b978ed71ad918a84 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.334897 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc (DEFAULT_TABLE table=default.Table_0b02bc193ec84c7abcc0754d885cfb89 [id=0e277dd692e842b28638b8675da40ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.342927 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c8286bc81dc24b5e8159136211d18799" } new_table_name: "default.Table_004cc703a3f541a9a31c0f4338618023" modify_external_catalogs: true
W20240507 08:11:52.346947 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 916e33e40ef44d06a359b82d8fe48157 consensus state has no leader
I20240507 08:11:52.350124 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b47f75826f1a465a9c38010bdf4d65ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.350550 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a749784e5d4446c4918d32b35efd5451" } new_table_name: "default.Table_c48da430b2554361ab7c682e78a6a54f" modify_external_catalogs: true
W20240507 08:11:52.350874 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b47f75826f1a465a9c38010bdf4d65ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.352043 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_556bd9f671734de4a0546ca45974a3ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.352478 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cb0dc1032c164b088e9d31013da8e43e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.352921 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_237e57be541d40359403f08fcc681dd8" } new_table_name: "default.Table_2ee2e89703be4de1a6f05d13e2af9abb" modify_external_catalogs: true
W20240507 08:11:52.353005 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb0dc1032c164b088e9d31013da8e43e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.353219 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b8c75caecb914fe2afb9c345bba1be57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.354094 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8c75caecb914fe2afb9c345bba1be57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.354298 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_71fb81b62c8746bf9376dd00c4250d3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.354956 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71fb81b62c8746bf9376dd00c4250d3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.352506 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_556bd9f671734de4a0546ca45974a3ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.357412 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.360489 5994 raft_consensus.cc:482] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:52.362766 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907b5cb0e7a54903abe0b0d0e8d617dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.369207 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:52.370200 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240507 08:11:52.370801 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8615f89e816e48fc8a058e5dc466918c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.374380 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:11:52.374569 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:11:52.376868 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b2de528d9fc046f183f70388cffab6c8 (DEFAULT_TABLE table=default.Table_8d15746afd6042b29e476bb2ecfa6c6e [id=317e03893acd43ef95533e458ed4346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.381600 5698 tablet_service.cc:1467] Processing CreateTablet for tablet d537bed29a2f47058255d2530b5c0d23 (DEFAULT_TABLE table=default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.382300 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.382922 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2de528d9fc046f183f70388cffab6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.391081 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 916e33e40ef44d06a359b82d8fe48157 (DEFAULT_TABLE table=default.Table_c93fda7fb0754945b3d3764e5698344e [id=14589abdd357409cbe9837a33208e00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.392086 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e4e33f52df432b8d043171d7232c2a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:52.393728 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf73c218b944679a12d769b1929e535 with delete_type TABLET_DATA_DELETED (replaced by tablet 169626f9a0bc4c6f9d8be11c0a802816) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.350093 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8881f6c0a43c45f6b13ffc2d61012180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.397230 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8881f6c0a43c45f6b13ffc2d61012180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.398669 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88a2cfb5f046434c8340234a447a7e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.399554 5974 raft_consensus.cc:482] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.411042 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:11:52.411232 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:11:52.344611 5553 tablet_service.cc:1467] Processing CreateTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (DEFAULT_TABLE table=default.Table_914ee05f51f14d84bfc1418f5a9cfb27 [id=70d5dddc40a3453bb64b8019c2a18046]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.335136 5549 tablet_service.cc:1467] Processing CreateTablet for tablet bde62374dc7f4a67a071b6b658e501c4 (DEFAULT_TABLE table=default.Table_923d5aad72744e84a98819df252a2e2c [id=d0a57f2571ca478084824a44922ac99b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.420061 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 907b5cb0e7a54903abe0b0d0e8d617dc consensus state has no leader
I20240507 08:11:52.420578 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_66abcf77e5b04c1ea97c7079c91bfada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.420944 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_21a782d34aee4be78034577651967f0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.421501 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 consensus state has no leader
I20240507 08:11:52.421936 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c83fcbd0cded4b18aa24693989f2f9ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.422430 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c83fcbd0cded4b18aa24693989f2f9ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.422966 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21a782d34aee4be78034577651967f0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.424253 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet add94bcf0c844cb78059b7240c7e28c0 consensus state has no leader
W20240507 08:11:52.427187 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66abcf77e5b04c1ea97c7079c91bfada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.431375 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.434876 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde62374dc7f4a67a071b6b658e501c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.435685 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 072702ef2a6d43248132da6aa3a2d493 (DEFAULT_TABLE table=default.Table_ce67a10a1e3f48a8b7225aad61e776e0 [id=3d99e1ae7dc34ac49843ce5f159f891e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.443099 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d537bed29a2f47058255d2530b5c0d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.444053 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 2612c023740644cb9564c44f7bfe8b61 (DEFAULT_TABLE table=default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.436106 5907 raft_consensus.cc:504] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.435513 5974 raft_consensus.cc:504] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } }
I20240507 08:11:52.449782 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 5089db17ca904a04878d83209001c54e (DEFAULT_TABLE table=default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.453444 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:52.453683 5444 raft_consensus.cc:3037] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.459769 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612c023740644cb9564c44f7bfe8b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.463028 5994 raft_consensus.cc:3037] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.467025 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089db17ca904a04878d83209001c54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.468645 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.468958 5547 tablet_service.cc:1467] Processing CreateTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (DEFAULT_TABLE table=default.Table_7004d9f9040b4d57abb409a848b45e48 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.470302 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c33710c6b55744c4ba0843c228abded3 (DEFAULT_TABLE table=default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.470768 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 364903504aa14d08b050a0cf83b82c71 (DEFAULT_TABLE table=default.Table_d9f6ed618e184cc6aece7c2d0dca9df8 [id=95b44cbcf26f489c8f4d8e07cc615776]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.471977 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet add94bcf0c844cb78059b7240c7e28c0 (table default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet add94bcf0c844cb78059b7240c7e28c0 consensus state has no leader
I20240507 08:11:52.472344 5841 ts_tablet_manager.cc:1888] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:52.472497 5841 log.cc:1196] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e8a046cce1f6467390071746a5e39fd9
I20240507 08:11:52.472939 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e4e33f52df432b8d043171d7232c2a with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa4ec7b93943e5bf8f9758b3bb71e0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.484580 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.487251 5841 ts_tablet_manager.cc:1909] T e8a046cce1f6467390071746a5e39fd9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:52.488087 5415 tablet_service.cc:1467] Processing CreateTablet for tablet b22334378eec4ab3885d0f9d2d812a3e (DEFAULT_TABLE table=default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.490054 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 973a199a533b4bbea3c2615acdbf95ca (DEFAULT_TABLE table=default.Table_d455ec93c6a1445a8948ee6973d6e4c2 [id=4ecddc64b3da4de5a70e7a211f9ea037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.490042 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b8044e4a0a41218f67513b14c0ebeb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:52.490088 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:52.490298 5719 raft_consensus.cc:2454] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:11:52.490710 5654 leader_election.cc:341] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.491011 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 84421.7us
I20240507 08:11:52.486353 5994 raft_consensus.cc:504] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.491180 5445 raft_consensus.cc:3037] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.492334 5840 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:52.492528 5840 ts_tablet_manager.cc:1367] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.260s user 0.005s sys 0.006s
I20240507 08:11:52.492969 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 169626f9a0bc4c6f9d8be11c0a802816 (DEFAULT_TABLE table=default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.493359 5840 raft_consensus.cc:348] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.493773 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 (DEFAULT_TABLE table=default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.494112 5427 tablet_service.cc:1467] Processing CreateTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 (DEFAULT_TABLE table=default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.495335 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed (DEFAULT_TABLE table=default.Table_a58b7d46a51f41049815a957ddac3f55 [id=fab372c29d234af7b0bfdae99be63919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.496354 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 107d9b3f95e14044854ad0bf91a268ee (DEFAULT_TABLE table=default.Table_30365e8917ec45c0afafa640e07f03dc [id=ccea3f017cc84cce95f3010023a76bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.496707 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22334378eec4ab3885d0f9d2d812a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.497861 5994 leader_election.cc:288] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.498128 5377 leader_election.cc:302] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.498548 5994 raft_consensus.cc:2781] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:52.501549 5974 leader_election.cc:288] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.502485 5994 raft_consensus.cc:482] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:52.502605 5994 raft_consensus.cc:3037] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.502015 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 34bebec088a34ebbb06d2e707d54da4c (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.506232 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107d9b3f95e14044854ad0bf91a268ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.509939 5445 raft_consensus.cc:2454] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:52.513412 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:52.513749 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:52.514125 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87e1f43426b14f98b985656f8d0d7141" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.514364 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.517222 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33710c6b55744c4ba0843c228abded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.518072 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 193ec0b463d04835b74df80229c85cde (DEFAULT_TABLE table=default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.519021 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.519973 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74342c8aee4017a212aa88a7f3dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.520931 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072702ef2a6d43248132da6aa3a2d493. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.528509 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.528939 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (DEFAULT_TABLE table=default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.531149 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd02a49359743c4b4e540ab0338d8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.532200 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 5554ebc621c24d459668f2cefc20b641 (DEFAULT_TABLE table=default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.534313 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 077ccc3946b34008bea7b80d8aee2662 (DEFAULT_TABLE table=default.Table_84d691a98c0d402fb6532f0e49f61488 [id=34d5ccf8d18945608504d42ba58ad664]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.535462 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba7c1aaf55d74c979ec1fc0a57f15f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.536152 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 104) took 1449 ms. Trace:
I20240507 08:11:52.540712 5995 raft_consensus.cc:482] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.540870 5995 raft_consensus.cc:504] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.490581 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:52.559831 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 364903504aa14d08b050a0cf83b82c71. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.561074 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193ec0b463d04835b74df80229c85cde. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.569835 5520 leader_election.cc:341] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 2cfa9a247eba402cb3f0b818ad9f837a
I20240507 08:11:52.580248 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc (DEFAULT_TABLE table=default.Table_0b02bc193ec84c7abcc0754d885cfb89 [id=0e277dd692e842b28638b8675da40ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.587185 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34bebec088a34ebbb06d2e707d54da4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.588656 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f2b43daab6b4deeac94fc6f1ff34a76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.594875 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a199a533b4bbea3c2615acdbf95ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.601280 5378 leader_election.cc:341] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.601559 5995 leader_election.cc:288] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.601835 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 19efea8905cf4e83a523c278b2f16dc7 (table default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:52.602250 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 81919c8f720f4c72bc2cc457b095859b (table default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]) successfully deleted
I20240507 08:11:52.602280 5444 raft_consensus.cc:2454] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:11:52.602646 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 98880a5ff9604cb4981c033c99e80be7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98880a5ff9604cb4981c033c99e80be7 already in progress: creating tablet
W20240507 08:11:52.603031 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f29568ebe4774662a8b49b7829cf30e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f29568ebe4774662a8b49b7829cf30e7 already in progress: creating tablet
I20240507 08:11:52.603996 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee3a6c4ac642a3921560ea13bc1341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.604025 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5cb62017acbd4472ac5b143eafb36ae9 (table default.Table_345d6952a3384ee29f957390161d28dc [id=f905dae7d04148bfab1cea83a820ceb0]) successfully deleted
I20240507 08:11:52.603343 5994 raft_consensus.cc:504] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.604209 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c68b6c08f00d46779a31f4e9eed99f9f (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
W20240507 08:11:52.604423 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 344bf4ab39f74accb383d695578de604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344bf4ab39f74accb383d695578de604 already in progress: creating tablet
W20240507 08:11:52.604653 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc because the tablet was not found. No further retry: Not found: Tablet not found: d9ff5e1b6a9a476c8295a0d0ee85aecc
W20240507 08:11:52.604833 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f922a7519084409db98693ee8797592b because the tablet was not found. No further retry: Not found: Tablet not found: f922a7519084409db98693ee8797592b
W20240507 08:11:52.605043 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ebc6cc72bb7e4721a8ab559102273db5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc6cc72bb7e4721a8ab559102273db5 already in progress: creating tablet
I20240507 08:11:52.603503 5841 rpcz_store.cc:276] 0507 08:11:51.066190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.066248 (+ 58us) service_pool.cc:225] Handling call
0507 08:11:51.525531 (+459283us) ts_tablet_manager.cc:1120] Deleting tablet e8a046cce1f6467390071746a5e39fd9
0507 08:11:51.525540 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:52.117262 (+591722us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.462752 (+345490us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.515910 (+ 53158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":459085}
I20240507 08:11:52.609061 5841 tablet_replica.cc:330] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:52.609313 5841 raft_consensus.cc:2229] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:52.609520 5841 raft_consensus.cc:2258] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:52.609941 5560 tablet_service.cc:1467] Processing CreateTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 (DEFAULT_TABLE table=default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.611500 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e8a046cce1f6467390071746a5e39fd9 (table default.Table_bd16315ec95647f699c27a2df5403d6e [id=3cfd0e33291941528c7b086eb06e6bd8]) successfully deleted
W20240507 08:11:52.611725 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e49dfd218a24464fbac39a3f4b477462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e49dfd218a24464fbac39a3f4b477462 already in progress: creating tablet
W20240507 08:11:52.612202 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5228590c8e7d4c34b3acab9eb16665d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5228590c8e7d4c34b3acab9eb16665d3 already in progress: creating tablet
W20240507 08:11:52.614436 5378 leader_election.cc:341] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.614792 5378 leader_election.cc:341] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.614877 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 077ccc3946b34008bea7b80d8aee2662. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.616521 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.617102 5994 leader_election.cc:288] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.617831 5558 tablet_service.cc:1467] Processing CreateTablet for tablet d537bed29a2f47058255d2530b5c0d23 (DEFAULT_TABLE table=default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.620532 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f6f62c56cb4c29bf05914ca461a559" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.621321 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.622112 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf73c218b944679a12d769b1929e535 with delete_type TABLET_DATA_DELETED (replaced by tablet 169626f9a0bc4c6f9d8be11c0a802816) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.623205 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 916e33e40ef44d06a359b82d8fe48157 (DEFAULT_TABLE table=default.Table_c93fda7fb0754945b3d3764e5698344e [id=14589abdd357409cbe9837a33208e00d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.623719 5838 tablet_replica.cc:330] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:52.623945 5838 raft_consensus.cc:2229] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:52.624629 5838 raft_consensus.cc:2258] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:52.625398 5838 ts_tablet_manager.cc:1875] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:52.626569 5839 tablet_bootstrap.cc:654] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:52.628322 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e4e33f52df432b8d043171d7232c2a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.629916 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e33e40ef44d06a359b82d8fe48157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.493561 5840 raft_consensus.cc:374] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.631467 5840 raft_consensus.cc:729] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.631758 5840 consensus_queue.cc:260] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.500279 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 916e33e40ef44d06a359b82d8fe48157. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.500339 5907 leader_election.cc:288] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.535014 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.513939 5720 raft_consensus.cc:2454] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:52.538240 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.538045 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:52.538748 5654 consensus_peers.cc:485] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:52.544181 5718 raft_consensus.cc:2379] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc665657d5b34fd7aec884e0f2668e11 in current term 1: Already voted for candidate 1cbb887175f345ada2433adaec906fd4 in this term.
W20240507 08:11:52.632761 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 116357us
I20240507 08:11:52.549681 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:52.632998 5654 leader_election.cc:341] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.633123 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 907b5cb0e7a54903abe0b0d0e8d617dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.551873 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f6f62c56cb4c29bf05914ca461a559" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:52.633389 5716 raft_consensus.cc:3037] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.545142 5656 leader_election.cc:302] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:52.633589 5654 leader_election.cc:341] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.633817 5654 leader_election.cc:341] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.555011 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 2612c023740644cb9564c44f7bfe8b61 (DEFAULT_TABLE table=default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.535041 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.563131 5837 ts_tablet_manager.cc:1888] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:52.634687 5837 log.cc:1196] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/81919c8f720f4c72bc2cc457b095859b
I20240507 08:11:52.635182 5837 ts_tablet_manager.cc:1909] T 81919c8f720f4c72bc2cc457b095859b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:52.567996 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 (DEFAULT_TABLE table=default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.636004 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b8044e4a0a41218f67513b14c0ebeb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:52.636200 5720 raft_consensus.cc:3037] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.636744 5716 raft_consensus.cc:2454] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:52.555048 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 5089db17ca904a04878d83209001c54e (DEFAULT_TABLE table=default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.579335 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e4e33f52df432b8d043171d7232c2a with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa4ec7b93943e5bf8f9758b3bb71e0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.639454 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:52.639916 5720 raft_consensus.cc:2454] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:52.556869 5688 tablet_service.cc:1467] Processing CreateTablet for tablet c33710c6b55744c4ba0843c228abded3 (DEFAULT_TABLE table=default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.587378 5686 tablet_service.cc:1467] Processing CreateTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 (DEFAULT_TABLE table=default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.589078 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 107d9b3f95e14044854ad0bf91a268ee (DEFAULT_TABLE table=default.Table_30365e8917ec45c0afafa640e07f03dc [id=ccea3f017cc84cce95f3010023a76bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.632337 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169626f9a0bc4c6f9d8be11c0a802816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.641762 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee3a6c4ac642a3921560ea13bc1341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.537843 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.645727 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612c023740644cb9564c44f7bfe8b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.646435 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089db17ca904a04878d83209001c54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.538144 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:52.635563 5654 consensus_peers.cc:485] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:52.647693 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5554ebc621c24d459668f2cefc20b641. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.650180 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1f9a56a95074e67bb33ceee1af48a18. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.652523 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33710c6b55744c4ba0843c228abded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.653301 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74342c8aee4017a212aa88a7f3dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.653738 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107d9b3f95e14044854ad0bf91a268ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.591392 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 973a199a533b4bbea3c2615acdbf95ca (DEFAULT_TABLE table=default.Table_d455ec93c6a1445a8948ee6973d6e4c2 [id=4ecddc64b3da4de5a70e7a211f9ea037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.593713 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b22334378eec4ab3885d0f9d2d812a3e (DEFAULT_TABLE table=default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.583213 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.662791 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d537bed29a2f47058255d2530b5c0d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.666392 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a199a533b4bbea3c2615acdbf95ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.666299 5559 tablet_service.cc:1467] Processing CreateTablet for tablet b2de528d9fc046f183f70388cffab6c8 (DEFAULT_TABLE table=default.Table_8d15746afd6042b29e476bb2ecfa6c6e [id=317e03893acd43ef95533e458ed4346a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.666702 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2de528d9fc046f183f70388cffab6c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.667519 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22334378eec4ab3885d0f9d2d812a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.628515 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed (DEFAULT_TABLE table=default.Table_a58b7d46a51f41049815a957ddac3f55 [id=fab372c29d234af7b0bfdae99be63919]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.672039 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd02a49359743c4b4e540ab0338d8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.673993 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b8044e4a0a41218f67513b14c0ebeb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.675279 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1063aa5c7014d84b66f74ce28e3d890" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:52.677083 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 160254us
I20240507 08:11:52.677148 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3e8b4d5d594f378451009c313f6bfa (DEFAULT_TABLE table=default.Table_4a583347a22943338c2cad0211f8f17e [id=06457cc0f83a470a95273688e948d21a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.677516 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3e8b4d5d594f378451009c313f6bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.677846 5656 leader_election.cc:302] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:52.678371 5891 raft_consensus.cc:2781] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:52.678963 5987 raft_consensus.cc:2781] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:52.679144 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44f6f62c56cb4c29bf05914ca461a559" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:52.679898 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:52.680342 5654 consensus_peers.cc:485] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:52.685840 5851 consensus_queue.cc:1035] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:52.687772 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9b8044e4a0a41218f67513b14c0ebeb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:52.688243 5517 leader_election.cc:302] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:11:52.688478 5517 raft_consensus.cc:2670] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:52.692440 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.692852 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 169626f9a0bc4c6f9d8be11c0a802816 (DEFAULT_TABLE table=default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.693261 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 169626f9a0bc4c6f9d8be11c0a802816. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.694589 5257 catalog_manager.cc:5484] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:52.695226 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet baa842552d00489998f83e68aa0d516b (table default.Table_5fc65aee40154e6296dc8b3f3076a4f3 [id=13e03112a7a641828153301cefc1774e]). Expected version 2 got 0
I20240507 08:11:52.696704 5408 tablet_service.cc:1467] Processing CreateTablet for tablet d45d3fbbe3764261a3a24523a77c6443 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.700600 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.700831 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.701037 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.701218 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.701403 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.705001 5377 leader_election.cc:302] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.705257 5978 consensus_queue.cc:1035] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.044s
I20240507 08:11:52.710034 5995 consensus_queue.cc:1035] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.072s
I20240507 08:11:52.711542 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 (DEFAULT_TABLE table=default.Table_556bd9f671734de4a0546ca45974a3ba [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.712046 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 5089db17ca904a04878d83209001c54e (DEFAULT_TABLE table=default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.712432 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5089db17ca904a04878d83209001c54e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.713706 5841 ts_tablet_manager.cc:1875] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:52.714501 6011 tablet.cc:1722] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:52.723119 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45d3fbbe3764261a3a24523a77c6443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.701565 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.724699 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 4716c1285e91432a92b25ab13ada2779 (DEFAULT_TABLE table=default.Table_b8c75caecb914fe2afb9c345bba1be57 [id=bd83f9b19311401a8ff3fd39e8c75e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.724776 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (DEFAULT_TABLE table=default.Table_8881f6c0a43c45f6b13ffc2d61012180 [id=33eb2f3eadb1433a833d373add05496c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.725845 5974 consensus_queue.cc:1035] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240507 08:11:52.727722 5840 ts_tablet_manager.cc:1398] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.235s user 0.000s sys 0.002s
I20240507 08:11:52.728323 5840 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:52.730156 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2cfa9a247eba402cb3f0b818ad9f837a with delete_type TABLET_DATA_DELETED (replaced by tablet 697611671ef64c53b0be4f5341b8f270) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:52.735181 5517 leader_election.cc:341] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 2cfa9a247eba402cb3f0b818ad9f837a
I20240507 08:11:52.736363 5553 tablet_service.cc:1467] Processing CreateTablet for tablet c33710c6b55744c4ba0843c228abded3 (DEFAULT_TABLE table=default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.744145 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 2612c023740644cb9564c44f7bfe8b61 (DEFAULT_TABLE table=default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.745992 5517 leader_election.cc:302] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:52.747546 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e5a3509f77064f24a47bc7d69395d0c7" } new_table_name: "default.Table_57ae6d7b295a4fb38adfb6579b01dc83" modify_external_catalogs: true
I20240507 08:11:52.747890 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc1d7e309946d783a524b61d5f1269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.748907 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716c1285e91432a92b25ab13ada2779. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.754202 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0ea74702c4b4514a31f0fad7e7a2f5b consensus state has no leader
I20240507 08:11:52.755267 5974 raft_consensus.cc:2726] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:52.755527 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e4e33f52df432b8d043171d7232c2a with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa4ec7b93943e5bf8f9758b3bb71e0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.756008 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:52.756443 5548 tablet_service.cc:1467] Processing CreateTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 (DEFAULT_TABLE table=default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.757992 5839 tablet_bootstrap.cc:492] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:11:52.757846 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 67137.3us
I20240507 08:11:52.758322 5377 leader_election.cc:302] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:52.759023 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 60bc7bc06e494adcb6a8641b50b90e6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60bc7bc06e494adcb6a8641b50b90e6c already in progress: creating tablet
I20240507 08:11:52.759163 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.760013 5377 leader_election.cc:341] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.760233 5377 leader_election.cc:302] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:52.761619 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 (DEFAULT_TABLE table=default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.762176 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 270a1fd15925416483c211021549315c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 270a1fd15925416483c211021549315c already in progress: creating tablet
I20240507 08:11:52.762534 5411 tablet_service.cc:1467] Processing CreateTablet for tablet c48da19c9b9e4207b267137d240adc5b (DEFAULT_TABLE table=default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.762883 5901 consensus_queue.cc:1035] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:52.763113 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (DEFAULT_TABLE table=default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.762890 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 1b19d8e28f164c588946ea5e81ec320e (DEFAULT_TABLE table=default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.763492 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.764751 5843 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.167s sys 0.246s
I20240507 08:11:52.765453 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5fc65aee40154e6296dc8b3f3076a4f3" } modify_external_catalogs: true
I20240507 08:11:52.765682 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5fc65aee40154e6296dc8b3f3076a4f3" } modify_external_catalogs: true
I20240507 08:11:52.767628 5840 tablet_bootstrap.cc:654] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:52.767774 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_af2b4d29f1e64f2d8e48d5b37c48318a" } modify_external_catalogs: true
I20240507 08:11:52.767972 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_af2b4d29f1e64f2d8e48d5b37c48318a" } modify_external_catalogs: true
I20240507 08:11:52.769284 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 193ec0b463d04835b74df80229c85cde (DEFAULT_TABLE table=default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.770994 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_014d10f31cfe4b05a42d8f9c3d3921e3" } modify_external_catalogs: true
I20240507 08:11:52.771178 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_014d10f31cfe4b05a42d8f9c3d3921e3" } modify_external_catalogs: true
I20240507 08:11:52.771224 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cc2b1c5d509b43afa9523de30d68a4d9" } modify_external_catalogs: true
I20240507 08:11:52.771435 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cc2b1c5d509b43afa9523de30d68a4d9" } modify_external_catalogs: true
I20240507 08:11:52.772444 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_edd5577d442c45a3b978ed71ad918a84" } new_table_name: "default.Table_722dbcaa48584561970775496bbde987" modify_external_catalogs: true
I20240507 08:11:52.773618 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet e0f3ee2b04de4cdf8a621b39f1476577 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.774320 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c290c4af761c45e885f1f4af9a48f4fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.774884 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c290c4af761c45e885f1f4af9a48f4fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.776063 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c58677b45a2b461490f3de5ca89c7b21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.776532 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c58677b45a2b461490f3de5ca89c7b21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.777863 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30365e8917ec45c0afafa640e07f03dc" } new_table_name: "default.Table_351291351f3c4972ac5f9cd896d68312" modify_external_catalogs: true
I20240507 08:11:52.778980 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet f968f122c39b4fd5b995e8a2848a52d4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.779516 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a2f27f59d07b4b23a4157ff49c53b62c" } new_table_name: "default.Table_f29e8299e1fc4d2ba2b6cd3036641274" modify_external_catalogs: true
I20240507 08:11:52.780766 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a0ac2efb18004de2977210793a4b8de7" } modify_external_catalogs: true
I20240507 08:11:52.781035 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a0ac2efb18004de2977210793a4b8de7" } modify_external_catalogs: true
I20240507 08:11:52.782028 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 (DEFAULT_TABLE table=default.Table_556bd9f671734de4a0546ca45974a3ba [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.782477 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc1d7e309946d783a524b61d5f1269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.772454 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet baa842552d00489998f83e68aa0d516b
I20240507 08:11:52.783941 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_931a381db9684935b68533da48152c99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.784067 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 4716c1285e91432a92b25ab13ada2779 (DEFAULT_TABLE table=default.Table_b8c75caecb914fe2afb9c345bba1be57 [id=bd83f9b19311401a8ff3fd39e8c75e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.784371 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (DEFAULT_TABLE table=default.Table_8881f6c0a43c45f6b13ffc2d61012180 [id=33eb2f3eadb1433a833d373add05496c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.784422 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_931a381db9684935b68533da48152c99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.784507 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 1b19d8e28f164c588946ea5e81ec320e (DEFAULT_TABLE table=default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.784437 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (DEFAULT_TABLE table=default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.784920 5682 tablet_service.cc:1467] Processing CreateTablet for tablet d45d3fbbe3764261a3a24523a77c6443 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.786422 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88a2cfb5f046434c8340234a447a7e65 consensus state has no leader
I20240507 08:11:52.786844 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6256978b02b54f189b2c9d7b524d6f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.786970 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_31ed6dfebe17497d9a90b837f72f1c2e" } new_table_name: "default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7" modify_external_catalogs: true
W20240507 08:11:52.787329 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6256978b02b54f189b2c9d7b524d6f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.788429 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 107d9b3f95e14044854ad0bf91a268ee consensus state has no leader
I20240507 08:11:52.788779 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_15145da393ce4d249d367dfdfc2efd4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.789235 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15145da393ce4d249d367dfdfc2efd4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.795778 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 812dd1ebd42f4e929838a0535c22ad69 consensus state has no leader
I20240507 08:11:52.796036 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet add94bcf0c844cb78059b7240c7e28c0 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.796510 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 34e4e33f52df432b8d043171d7232c2a from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.798265 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 120ebfd34a9e4d73a215cb9206013728 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:52.804490 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2612c023740644cb9564c44f7bfe8b61. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.805122 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6bf34089e5ec419f97c0da9ec42d7eec
W20240507 08:11:52.805850 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5ca8fc2c7f3849c380fea56862ffc26e consensus state has no leader
I20240507 08:11:52.806273 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet baa842552d00489998f83e68aa0d516b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.803964 5681 tablet_service.cc:1467] Processing CreateTablet for tablet c48da19c9b9e4207b267137d240adc5b (DEFAULT_TABLE table=default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.808185 5688 tablet_service.cc:1467] Processing CreateTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba (DEFAULT_TABLE table=default.Table_66abcf77e5b04c1ea97c7079c91bfada [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.808322 5689 tablet_service.cc:1467] Processing CreateTablet for tablet dfa14901cdf14b72bea365b04ad521bd (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.810509 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193ec0b463d04835b74df80229c85cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.810827 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa14901cdf14b72bea365b04ad521bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.811044 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7df45c39d1b490db97c72a0eb6cf3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.811298 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48da19c9b9e4207b267137d240adc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.811677 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.811990 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.813702 5839 ts_tablet_manager.cc:1367] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.552s user 0.002s sys 0.003s
I20240507 08:11:52.813781 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45d3fbbe3764261a3a24523a77c6443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.821666 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48da19c9b9e4207b267137d240adc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.822952 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 107d9b3f95e14044854ad0bf91a268ee (DEFAULT_TABLE table=default.Table_30365e8917ec45c0afafa640e07f03dc [id=ccea3f017cc84cce95f3010023a76bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.822484 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 973a199a533b4bbea3c2615acdbf95ca (DEFAULT_TABLE table=default.Table_d455ec93c6a1445a8948ee6973d6e4c2 [id=4ecddc64b3da4de5a70e7a211f9ea037]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.814574 5839 raft_consensus.cc:348] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.824352 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b19d8e28f164c588946ea5e81ec320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.826678 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716c1285e91432a92b25ab13ada2779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.827772 5839 raft_consensus.cc:374] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.827884 5839 raft_consensus.cc:729] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.828303 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 5782cfa5eca74c19a2e724a9124597f2 (DEFAULT_TABLE table=default.Table_5e90ac34a69740dca680fc75e5738dcd [id=c6a13f9902b0424ca8df2184779a407c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.815045 5840 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:52.828743 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5782cfa5eca74c19a2e724a9124597f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.829799 5987 raft_consensus.cc:686] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:52.830101 5907 raft_consensus.cc:482] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.830118 5839 consensus_queue.cc:260] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.830684 5839 ts_tablet_manager.cc:1398] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240507 08:11:52.832599 5839 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:52.832954 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 124) took 1352 ms. Trace:
I20240507 08:11:52.833096 5987 consensus_queue.cc:237] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.833472 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc74342c8aee4017a212aa88a7f3dc06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.838492 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973a199a533b4bbea3c2615acdbf95ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.837102 5840 ts_tablet_manager.cc:1367] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.001s
I20240507 08:11:52.837039 5891 raft_consensus.cc:686] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:52.841187 5907 raft_consensus.cc:504] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.843670 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b (DEFAULT_TABLE table=default.Table_c83fcbd0cded4b18aa24693989f2f9ef [id=f21341e3ea0a49ebacc9309604184464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.844105 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89db2847c09b4c74b809bd13f78d0b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.845997 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 191adef34efa4c3999d4642a80177294
I20240507 08:11:52.846797 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet add94bcf0c844cb78059b7240c7e28c0
I20240507 08:11:52.847219 5263 catalog_manager.cc:6554] ReplaceTablet: tablet add94bcf0c844cb78059b7240c7e28c0 deleted and replaced by tablet b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:11:52.847955 5563 tablet_service.cc:1467] Processing CreateTablet for tablet b22334378eec4ab3885d0f9d2d812a3e (DEFAULT_TABLE table=default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.849710 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 80d82625304942d38b0261f95e235a41
I20240507 08:11:52.850160 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33710c6b55744c4ba0843c228abded3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.850183 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf2df0d3bd46268e22713fcc135a74" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.850718 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 107d9b3f95e14044854ad0bf91a268ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.850996 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b22334378eec4ab3885d0f9d2d812a3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.854001 5978 raft_consensus.cc:2781] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:52.854095 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b19d8e28f164c588946ea5e81ec320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.854148 5978 raft_consensus.cc:482] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:52.858991 5995 raft_consensus.cc:2781] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:52.859231 5995 raft_consensus.cc:686] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:52.864084 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91ee3a6c4ac642a3921560ea13bc1341. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.864491 5839 tablet_bootstrap.cc:654] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:52.865207 5994 raft_consensus.cc:2781] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:52.866065 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed (DEFAULT_TABLE table=default.Table_a58b7d46a51f41049815a957ddac3f55 [id=fab372c29d234af7b0bfdae99be63919]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:52.866181 5843 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.574s user 0.167s sys 0.246s
I20240507 08:11:52.868167 5978 raft_consensus.cc:3037] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:52.870754 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d537bed29a2f47058255d2530b5c0d23
I20240507 08:11:52.874727 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet baa842552d00489998f83e68aa0d516b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:52.874958 5378 leader_election.cc:341] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.875216 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 112810us
W20240507 08:11:52.875478 5378 leader_election.cc:341] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.875844 5378 leader_election.cc:341] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:52.876225 5378 consensus_peers.cc:485] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:52.876418 5378 leader_election.cc:341] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.876564 5378 leader_election.cc:302] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:52.877142 5378 consensus_peers.cc:485] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:52.877475 5378 leader_election.cc:341] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.877723 5901 raft_consensus.cc:2726] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:52.877948 5412 tablet_service.cc:1467] Processing CreateTablet for tablet dfa14901cdf14b72bea365b04ad521bd (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.878455 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa14901cdf14b72bea365b04ad521bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.878635 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 5782cfa5eca74c19a2e724a9124597f2 (DEFAULT_TABLE table=default.Table_5e90ac34a69740dca680fc75e5738dcd [id=c6a13f9902b0424ca8df2184779a407c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.879060 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5782cfa5eca74c19a2e724a9124597f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.880050 5426 tablet_service.cc:1467] Processing CreateTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba (DEFAULT_TABLE table=default.Table_66abcf77e5b04c1ea97c7079c91bfada [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.881181 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7df45c39d1b490db97c72a0eb6cf3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.864151 5995 consensus_queue.cc:237] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:52.882938 5378 leader_election.cc:341] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.883121 5378 leader_election.cc:302] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:52.883540 5378 leader_election.cc:341] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.883955 5901 raft_consensus.cc:2726] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:52.886638 5994 raft_consensus.cc:686] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:52.887954 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 193ec0b463d04835b74df80229c85cde (DEFAULT_TABLE table=default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.889674 5557 tablet_service.cc:1467] Processing CreateTablet for tablet c48da19c9b9e4207b267137d240adc5b (DEFAULT_TABLE table=default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.890972 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b (DEFAULT_TABLE table=default.Table_c83fcbd0cded4b18aa24693989f2f9ef [id=f21341e3ea0a49ebacc9309604184464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.892158 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89db2847c09b4c74b809bd13f78d0b8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.892261 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cd02a49359743c4b4e540ab0338d8ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.892328 5843 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:52.893023 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 193ec0b463d04835b74df80229c85cde. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.896162 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c48da19c9b9e4207b267137d240adc5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.900086 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.901635 5843 ts_tablet_manager.cc:1367] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.791s user 0.168s sys 0.248s
I20240507 08:11:52.901674 5901 raft_consensus.cc:482] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:52.901835 5901 raft_consensus.cc:504] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.902175 6012 tablet.cc:1722] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:52.906112 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 (DEFAULT_TABLE table=default.Table_556bd9f671734de4a0546ca45974a3ba [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.906669 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fcc1d7e309946d783a524b61d5f1269. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.913458 5837 rpcz_store.cc:276] 0507 08:11:51.478241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.482019 (+ 3778us) service_pool.cc:225] Handling call
0507 08:11:52.081117 (+599098us) ts_tablet_manager.cc:1120] Deleting tablet 81919c8f720f4c72bc2cc457b095859b
0507 08:11:52.081128 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:52.360655 (+279527us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.559417 (+198762us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.830400 (+270983us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43569,"tablet-delete.queue_time_us":598909}
I20240507 08:11:52.914016 5837 tablet_replica.cc:330] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:52.914228 5837 raft_consensus.cc:2229] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:52.918681 5563 tablet_service.cc:1467] Processing CreateTablet for tablet d45d3fbbe3764261a3a24523a77c6443 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.919186 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45d3fbbe3764261a3a24523a77c6443. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.920377 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (DEFAULT_TABLE table=default.Table_8881f6c0a43c45f6b13ffc2d61012180 [id=33eb2f3eadb1433a833d373add05496c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.921752 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.928189 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f341e48a8269452aae35ac68fc04df21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.928826 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f341e48a8269452aae35ac68fc04df21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.935899 5378 consensus_peers.cc:485] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:52.941234 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.941195 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f968f122c39b4fd5b995e8a2848a52d4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:52.943048 5837 raft_consensus.cc:2258] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:52.944154 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 81919c8f720f4c72bc2cc457b095859b (table default.Table_b7024e203f50440095d2cf53ca83c775 [id=9eaa97919d7a4c4dac0c4ccd7867d904]) successfully deleted
I20240507 08:11:52.946807 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_420e2e62b141454298f31e925f51f2d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.947353 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_420e2e62b141454298f31e925f51f2d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.942955 5907 leader_election.cc:288] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:52.948535 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet add94bcf0c844cb78059b7240c7e28c0 with delete_type TABLET_DATA_DELETED (replaced by tablet b7947ca2d8324dbb9c909ec50852fed8) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.948809 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d82625304942d38b0261f95e235a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:52.952077 5654 leader_election.cc:341] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.953893 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 9e447562e82347a584e39586500c27c2 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.954668 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 7027b4cefa4d41659f819ee4326470c4 (DEFAULT_TABLE table=default.Table_15145da393ce4d249d367dfdfc2efd4f [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.955330 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c (DEFAULT_TABLE table=default.Table_931a381db9684935b68533da48152c99 [id=b1f2776824d24099bbc0110c3da97c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.955538 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 4716c1285e91432a92b25ab13ada2779 (DEFAULT_TABLE table=default.Table_b8c75caecb914fe2afb9c345bba1be57 [id=bd83f9b19311401a8ff3fd39e8c75e8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.955940 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4716c1285e91432a92b25ab13ada2779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.956735 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.949539 5891 consensus_queue.cc:237] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:52.964283 5837 ts_tablet_manager.cc:1875] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:52.945120 5994 consensus_queue.cc:237] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.966516 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:52.967047 5901 leader_election.cc:288] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:52.965849 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.967344 5843 raft_consensus.cc:348] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.967715 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet add94bcf0c844cb78059b7240c7e28c0 with delete_type TABLET_DATA_DELETED (replaced by tablet b7947ca2d8324dbb9c909ec50852fed8) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:52.969857 5378 leader_election.cc:341] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.970774 5843 raft_consensus.cc:374] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:52.970906 5843 raft_consensus.cc:729] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:52.971199 5843 consensus_queue.cc:260] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.971972 5843 ts_tablet_manager.cc:1398] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.070s user 0.001s sys 0.002s
I20240507 08:11:52.975591 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6b76dffef8004978b959721044683956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:52.975750 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1ccda05e59e0486b8a26054c5514666f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.976341 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ccda05e59e0486b8a26054c5514666f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.976704 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ce35197b07dc454c82587e250579e404"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:52.977321 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce35197b07dc454c82587e250579e404 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:52.977465 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b76dffef8004978b959721044683956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:52.969434 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf2df0d3bd46268e22713fcc135a74" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:52.981077 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 168e49ae24314a7aae2291485ddeda44 (DEFAULT_TABLE table=default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.981693 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e447562e82347a584e39586500c27c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.982069 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e49ae24314a7aae2291485ddeda44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.982213 5690 tablet_service.cc:1467] Processing CreateTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.982676 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7947ca2d8324dbb9c909ec50852fed8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:52.985823 5656 leader_election.cc:341] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:52.985998 5656 leader_election.cc:302] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:52.986641 5851 raft_consensus.cc:2726] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:52.986841 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.987466 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c978ec8bca4aa5b2ffaf22c3290747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.987883 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ce67a10a1e3f48a8b7225aad61e776e0" } new_table_name: "default.Table_c6bf901b2c4a491e8aa4c357670a5d08" modify_external_catalogs: true
I20240507 08:11:52.987474 5907 consensus_queue.cc:1035] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20240507 08:11:52.998420 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027b4cefa4d41659f819ee4326470c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.999053 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a425a90f0449bcaaacb4c3f653d26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:52.999420 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d82625304942d38b0261f95e235a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.999751 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 (DEFAULT_TABLE table=default.Table_f341e48a8269452aae35ac68fc04df21 [id=4851deb2bcf2430cb22de846f4410896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:52.999716 5840 raft_consensus.cc:348] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:52.999967 5840 raft_consensus.cc:374] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.000064 5840 raft_consensus.cc:729] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.005086 5840 consensus_queue.cc:260] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.005861 5840 ts_tablet_manager.cc:1398] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.166s user 0.002s sys 0.001s
I20240507 08:11:53.006207 5840 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.007995 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3" } modify_external_catalogs: true
I20240507 08:11:53.008055 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f341e48a8269452aae35ac68fc04df21" } new_table_name: "default.Table_77b68877a0464350bac7b63beb821606" modify_external_catalogs: true
I20240507 08:11:53.008205 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3" } modify_external_catalogs: true
I20240507 08:11:53.008648 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_914ee05f51f14d84bfc1418f5a9cfb27" } new_table_name: "default.Table_4994af241b3b4efe9518adcaa62fb422" modify_external_catalogs: true
I20240507 08:11:53.009788 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7959240cfedd4e58940d3093998d84c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.010294 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7959240cfedd4e58940d3093998d84c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.010442 5840 tablet_bootstrap.cc:654] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.011451 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_15145da393ce4d249d367dfdfc2efd4f" } new_table_name: "default.Table_675dec7414094d999f135b996112021f" modify_external_catalogs: true
I20240507 08:11:53.007006 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 168e49ae24314a7aae2291485ddeda44 (DEFAULT_TABLE table=default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.007252 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 9e447562e82347a584e39586500c27c2 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.018097 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:52.967873 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f968f122c39b4fd5b995e8a2848a52d4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:53.025671 5843 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:53.025799 5425 tablet_service.cc:1467] Processing CreateTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.017827 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 7027b4cefa4d41659f819ee4326470c4 (DEFAULT_TABLE table=default.Table_15145da393ce4d249d367dfdfc2efd4f [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.030782 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070fdbf0b00b4582a184729c2f2a4b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.033010 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c (DEFAULT_TABLE table=default.Table_931a381db9684935b68533da48152c99 [id=b1f2776824d24099bbc0110c3da97c2e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:53.033370 5443 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.539s user 0.035s sys 0.067s
W20240507 08:11:53.033470 5443 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.035s sys 0.068s
I20240507 08:11:53.038621 6011 tablet.cc:1717] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Skipping requested alter to schema version 4, tablet already version 4
I20240507 08:11:53.039170 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_53e17078e5714619b86165f5e3f636ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.039724 5377 leader_election.cc:341] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.040122 6011 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:55948 (request call id 186) took 1006 ms. Trace:
W20240507 08:11:53.040868 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53e17078e5714619b86165f5e3f636ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.040251 6011 rpcz_store.cc:276] 0507 08:11:52.033994 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.394464 (+360470us) service_pool.cc:225] Handling call
0507 08:11:53.040047 (+645583us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:11:52.676302 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0507 08:11:52.676412 (+ 110us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0507 08:11:52.721732 (+ 45320us) alter_schema_op.cc:59] Acquired schema lock
0507 08:11:52.721743 (+ 11us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0507 08:11:52.721789 (+ 46us) alter_schema_op.cc:119] START. Timestamp: 1715069512721725
0507 08:11:52.721792 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:11:52.722001 (+ 209us) log.cc:842] Serialized 173 byte log entry
0507 08:11:53.033087 (+311086us) op_driver.cc:444] REPLICATION: finished
0507 08:11:53.033180 (+ 93us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0507 08:11:53.039951 (+ 6771us) log.cc:842] Serialized 94 byte log entry
0507 08:11:53.040010 (+ 59us) alter_schema_op.cc:65] Released schema lock
0507 08:11:53.040017 (+ 7us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":30,"prepare.queue_time_us":239263,"prepare.run_cpu_time_us":796,"prepare.run_wall_time_us":46141,"raft.queue_time_us":230,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":311265}]]}
I20240507 08:11:53.044497 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7947ca2d8324dbb9c909ec50852fed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.049511 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.055827 5377 leader_election.cc:302] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:53.056156 5901 raft_consensus.cc:2726] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:53.056411 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e49ae24314a7aae2291485ddeda44. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.062974 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7027b4cefa4d41659f819ee4326470c4 consensus state has no leader
I20240507 08:11:53.063663 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_da387f5ea4c142428110959beb6cf669"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.064430 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da387f5ea4c142428110959beb6cf669 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.066322 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c84a2018f1c442e2a4dfc1c6b9c5ceef consensus state has no leader
W20240507 08:11:53.066622 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 072702ef2a6d43248132da6aa3a2d493 consensus state has no leader
I20240507 08:11:53.066702 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f68ddcc012de4bd9ba219332fbfa8760" } modify_external_catalogs: true
I20240507 08:11:53.066959 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f68ddcc012de4bd9ba219332fbfa8760" } modify_external_catalogs: true
I20240507 08:11:53.067023 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_351291351f3c4972ac5f9cd896d68312" } modify_external_catalogs: true
I20240507 08:11:53.067221 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_351291351f3c4972ac5f9cd896d68312" } modify_external_catalogs: true
I20240507 08:11:53.066136 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6a9d7d23343543d9aa714b9fb7c0c006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.067891 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 070fdbf0b00b4582a184729c2f2a4b69 consensus state has no leader
W20240507 08:11:53.067986 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a9d7d23343543d9aa714b9fb7c0c006 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.068353 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d455ec93c6a1445a8948ee6973d6e4c2" } new_table_name: "default.Table_964368f3c7824528b36f3fb37bb2e215" modify_external_catalogs: true
I20240507 08:11:53.070181 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e" } modify_external_catalogs: true
I20240507 08:11:53.070411 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e" } modify_external_catalogs: true
I20240507 08:11:53.070839 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c207fdb532c4420b8289d1d9959722dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.071450 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c207fdb532c4420b8289d1d9959722dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.071799 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bde62374dc7f4a67a071b6b658e501c4
I20240507 08:11:53.076362 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.079268 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6348e5422e1d4fedaac4c6084f846b96" } modify_external_catalogs: true
I20240507 08:11:53.079471 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6348e5422e1d4fedaac4c6084f846b96" } modify_external_catalogs: true
I20240507 08:11:53.102952 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:53.103082 5978 raft_consensus.cc:504] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:53.104179 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 973a199a533b4bbea3c2615acdbf95ca consensus state has no leader
I20240507 08:11:53.117234 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a425a90f0449bcaaacb4c3f653d26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.120913 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 (DEFAULT_TABLE table=default.Table_f341e48a8269452aae35ac68fc04df21 [id=4851deb2bcf2430cb22de846f4410896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.121176 5837 ts_tablet_manager.cc:1888] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.121433 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 1b19d8e28f164c588946ea5e81ec320e (DEFAULT_TABLE table=default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.121860 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b19d8e28f164c588946ea5e81ec320e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.125567 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (DEFAULT_TABLE table=default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.131213 6012 tablet.cc:1717] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Skipping requested alter to schema version 4, tablet already version 4
I20240507 08:11:53.133000 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d10b38120dfe4b9ab3c39cf584e4ff13
I20240507 08:11:53.135267 5978 leader_election.cc:288] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:53.138742 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.139014 5837 log.cc:1196] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d6d9ab293113410c857b5de903e21eb5
I20240507 08:11:53.138995 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7f36bf46aad74e46a282d6d05585b60d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.139582 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f36bf46aad74e46a282d6d05585b60d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.139621 5837 ts_tablet_manager.cc:1909] T d6d9ab293113410c857b5de903e21eb5 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.140403 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 122) took 1662 ms. Trace:
I20240507 08:11:53.140488 5837 rpcz_store.cc:276] 0507 08:11:51.477485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.482304 (+ 4819us) service_pool.cc:225] Handling call
0507 08:11:52.913948 (+1431644us) ts_tablet_manager.cc:1120] Deleting tablet d6d9ab293113410c857b5de903e21eb5
0507 08:11:52.913961 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:52.993359 (+ 79398us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.121065 (+127706us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.140349 (+ 19284us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13777,"tablet-delete.queue_time_us":1431501}
I20240507 08:11:53.141389 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e447562e82347a584e39586500c27c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.141857 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027b4cefa4d41659f819ee4326470c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.142113 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070fdbf0b00b4582a184729c2f2a4b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.142702 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c978ec8bca4aa5b2ffaf22c3290747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.144238 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.149291 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e4e33f52df432b8d043171d7232c2a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:53.150187 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d6d9ab293113410c857b5de903e21eb5 (table default.Table_cfeaf327b3644adf98fe2ee6d1a99ecc [id=8f0ba3f54ceb4158a2cb810f6a733ea3]) successfully deleted
I20240507 08:11:53.150688 5837 tablet_replica.cc:330] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.151358 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet baa842552d00489998f83e68aa0d516b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.152302 5551 tablet_service.cc:1467] Processing CreateTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba (DEFAULT_TABLE table=default.Table_66abcf77e5b04c1ea97c7079c91bfada [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.152853 5837 raft_consensus.cc:2229] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.153065 5837 raft_consensus.cc:2258] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.153646 5837 ts_tablet_manager.cc:1875] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.159365 5684 tablet_service.cc:1467] Processing CreateTablet for tablet b4504b1ed3d449619926fe80469d9790 (DEFAULT_TABLE table=default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.159559 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7df45c39d1b490db97c72a0eb6cf3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.162986 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.166066 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4504b1ed3d449619926fe80469d9790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.169611 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:53.169827 5378 leader_election.cc:341] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.170475 6011 tablet.cc:1717] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Skipping requested alter to schema version 4, tablet already version 4
I20240507 08:11:53.171588 6011 tablet.cc:1722] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:53.176540 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 107d9b3f95e14044854ad0bf91a268ee
I20240507 08:11:53.177609 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 193ec0b463d04835b74df80229c85cde
I20240507 08:11:53.179705 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.185710 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.187417 5843 tablet_bootstrap.cc:654] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.190642 5423 tablet_service.cc:1467] Processing CreateTablet for tablet b4504b1ed3d449619926fe80469d9790 (DEFAULT_TABLE table=default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.191495 5840 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.191673 5840 ts_tablet_manager.cc:1367] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.186s user 0.005s sys 0.002s
I20240507 08:11:53.192588 5840 raft_consensus.cc:348] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.192807 5840 raft_consensus.cc:374] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.192898 5840 raft_consensus.cc:729] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.193174 5840 consensus_queue.cc:260] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.193810 5840 ts_tablet_manager.cc:1398] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:53.194167 5840 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.199581 5840 tablet_bootstrap.cc:654] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.199843 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.199893 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.203826 5564 tablet_service.cc:1467] Processing CreateTablet for tablet dfa14901cdf14b72bea365b04ad521bd (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.205632 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.207065 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 7027b4cefa4d41659f819ee4326470c4 (DEFAULT_TABLE table=default.Table_15145da393ce4d249d367dfdfc2efd4f [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.207391 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4504b1ed3d449619926fe80469d9790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.208052 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]). Expected version 2 got 0
I20240507 08:11:53.208328 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:53.209887 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.210644 5837 ts_tablet_manager.cc:1888] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.210763 5837 log.cc:1196] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/a485c1dc1cae40fcadce8ce75a5607b2
I20240507 08:11:53.211151 5837 ts_tablet_manager.cc:1909] T a485c1dc1cae40fcadce8ce75a5607b2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.212000 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 118) took 1735 ms. Trace:
I20240507 08:11:53.212178 5837 rpcz_store.cc:276] 0507 08:11:51.476962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.509905 (+ 32943us) service_pool.cc:225] Handling call
0507 08:11:53.150590 (+1640685us) ts_tablet_manager.cc:1120] Deleting tablet a485c1dc1cae40fcadce8ce75a5607b2
0507 08:11:53.150604 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.196071 (+ 45467us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.210503 (+ 14432us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.211956 (+ 1453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640428}
I20240507 08:11:53.212669 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet a485c1dc1cae40fcadce8ce75a5607b2 (table default.Table_a873e078e67e44b4871044bbe3846aba [id=4de4049e60cf448598c3ca2e37a55277]) successfully deleted
I20240507 08:11:53.213044 5837 tablet_replica.cc:330] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:11:53.213097 5654 consensus_peers.cc:485] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:53.213511 5837 raft_consensus.cc:2229] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.214030 5837 pending_rounds.cc:62] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:53.214004 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c (DEFAULT_TABLE table=default.Table_931a381db9684935b68533da48152c99 [id=b1f2776824d24099bbc0110c3da97c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.214941 5837 pending_rounds.cc:68] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924717730238464 op_type: NO_OP noop_request { }
I20240507 08:11:53.215898 5837 raft_consensus.cc:2866] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:53.216221 5837 raft_consensus.cc:2258] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.217379 5837 ts_tablet_manager.cc:1875] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.221731 5698 tablet_service.cc:1467] Processing CreateTablet for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (DEFAULT_TABLE table=default.Table_da387f5ea4c142428110959beb6cf669 [id=26bfe0dbbeca415eb1900ef1ab5bd316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.222254 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2604a090c94c78bd4f7a1dea6f81b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.222872 5558 tablet_service.cc:1467] Processing CreateTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.223639 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfa14901cdf14b72bea365b04ad521bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.224493 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 5782cfa5eca74c19a2e724a9124597f2 (DEFAULT_TABLE table=default.Table_5e90ac34a69740dca680fc75e5738dcd [id=c6a13f9902b0424ca8df2184779a407c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.231315 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7027b4cefa4d41659f819ee4326470c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.232960 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9ced879f35e14106b2609c78b7f24002 (DEFAULT_TABLE table=default.Table_6a9d7d23343543d9aa714b9fb7c0c006 [id=aa09d786696945d5b4060819737a6e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.233453 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ced879f35e14106b2609c78b7f24002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.236629 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 (DEFAULT_TABLE table=default.Table_f341e48a8269452aae35ac68fc04df21 [id=4851deb2bcf2430cb22de846f4410896]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.237057 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5782cfa5eca74c19a2e724a9124597f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.238252 5837 ts_tablet_manager.cc:1888] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.239029 5837 log.cc:1196] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/98880a5ff9604cb4981c033c99e80be7
I20240507 08:11:53.239694 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 (DEFAULT_TABLE table=default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.240113 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4fa2d0d8a1a9482599a4d430029b2b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.240423 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87377661d9374cef8ab1a97042b7f1b9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.240794 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fa2d0d8a1a9482599a4d430029b2b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.241732 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet add94bcf0c844cb78059b7240c7e28c0 with delete_type TABLET_DATA_DELETED (replaced by tablet b7947ca2d8324dbb9c909ec50852fed8) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.241988 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.242173 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.242313 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.242632 5549 tablet_service.cc:1467] Processing CreateTablet for tablet b4504b1ed3d449619926fe80469d9790 (DEFAULT_TABLE table=default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.243710 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d82625304942d38b0261f95e235a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.243937 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.244673 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.245255 5547 tablet_service.cc:1467] Processing CreateTablet for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (DEFAULT_TABLE table=default.Table_da387f5ea4c142428110959beb6cf669 [id=26bfe0dbbeca415eb1900ef1ab5bd316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.246026 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a425a90f0449bcaaacb4c3f653d26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.247270 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.248082 5837 ts_tablet_manager.cc:1909] T 98880a5ff9604cb4981c033c99e80be7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.248108 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 9e447562e82347a584e39586500c27c2 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.250402 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 9ced879f35e14106b2609c78b7f24002 (DEFAULT_TABLE table=default.Table_6a9d7d23343543d9aa714b9fb7c0c006 [id=aa09d786696945d5b4060819737a6e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.250897 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.251340 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 117) took 1782 ms. Trace:
I20240507 08:11:53.253207 5692 tablet_service.cc:1467] Processing CreateTablet for tablet c707458d1c094d87b2204683b28b49e4 (DEFAULT_TABLE table=default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.254050 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.254282 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 835d7b779d724df4863bb13a5140dbf0 (DEFAULT_TABLE table=default.Table_1ccda05e59e0486b8a26054c5514666f [id=83f625713cda401f90e513ddef199bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.254441 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 (DEFAULT_TABLE table=default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.254658 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d7b779d724df4863bb13a5140dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.256472 6012 tablet.cc:1717] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Skipping requested alter to schema version 4, tablet already version 4
I20240507 08:11:53.257764 5413 tablet_service.cc:1467] Processing CreateTablet for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (DEFAULT_TABLE table=default.Table_da387f5ea4c142428110959beb6cf669 [id=26bfe0dbbeca415eb1900ef1ab5bd316]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.257882 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 81a1a42a22704974b1ef6cbd17d91854 (DEFAULT_TABLE table=default.Table_7959240cfedd4e58940d3093998d84c1 [id=9af183d91ab34faebbe34ea8c4ad4196]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.258283 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a1a42a22704974b1ef6cbd17d91854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.258970 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b (DEFAULT_TABLE table=default.Table_c83fcbd0cded4b18aa24693989f2f9ef [id=f21341e3ea0a49ebacc9309604184464]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.262080 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e555a89d340448bc852f2441ca850213 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.262467 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555a89d340448bc852f2441ca850213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.263620 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.264022 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.264640 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 9ced879f35e14106b2609c78b7f24002 (DEFAULT_TABLE table=default.Table_6a9d7d23343543d9aa714b9fb7c0c006 [id=aa09d786696945d5b4060819737a6e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.277756 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 835d7b779d724df4863bb13a5140dbf0 (DEFAULT_TABLE table=default.Table_1ccda05e59e0486b8a26054c5514666f [id=83f625713cda401f90e513ddef199bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.276368 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 (DEFAULT_TABLE table=default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.278754 5408 tablet_service.cc:1467] Processing CreateTablet for tablet c707458d1c094d87b2204683b28b49e4 (DEFAULT_TABLE table=default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.279043 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 81a1a42a22704974b1ef6cbd17d91854 (DEFAULT_TABLE table=default.Table_7959240cfedd4e58940d3093998d84c1 [id=9af183d91ab34faebbe34ea8c4ad4196]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.279964 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e34782f4ad6e45c8a22146e8d9204e89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.280148 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_675dec7414094d999f135b996112021f" } new_table_name: "default.Table_18cce6f7f33d4b8e92f4c21a57a93eae" modify_external_catalogs: true
W20240507 08:11:53.280794 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e34782f4ad6e45c8a22146e8d9204e89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.280197 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ced879f35e14106b2609c78b7f24002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.281909 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4ea4bbd2ff0a49a89be1371511b5986c" } new_table_name: "default.Table_4c19e6eecd154d29a5b2cc7dae88a82a" modify_external_catalogs: true
I20240507 08:11:53.282974 5410 tablet_service.cc:1467] Processing CreateTablet for tablet e555a89d340448bc852f2441ca850213 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.284703 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 168e49ae24314a7aae2291485ddeda44 (DEFAULT_TABLE table=default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.285146 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a1a42a22704974b1ef6cbd17d91854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.286245 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555a89d340448bc852f2441ca850213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.286381 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f29e8299e1fc4d2ba2b6cd3036641274" } modify_external_catalogs: true
I20240507 08:11:53.286566 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f29e8299e1fc4d2ba2b6cd3036641274" } modify_external_catalogs: true
I20240507 08:11:53.287077 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d9f6ed618e184cc6aece7c2d0dca9df8" } new_table_name: "default.Table_29880271f0474162818f1334e42c2a3d" modify_external_catalogs: true
I20240507 08:11:53.287328 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c076159e489f4d5a97e67533975f0a43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.289362 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2604a090c94c78bd4f7a1dea6f81b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.289784 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c076159e489f4d5a97e67533975f0a43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.290333 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.290936 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c707458d1c094d87b2204683b28b49e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.291482 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.291610 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_77b68877a0464350bac7b63beb821606" } new_table_name: "default.Table_b782575cf75a40b797e243b6a4059080" modify_external_catalogs: true
I20240507 08:11:53.291872 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87377661d9374cef8ab1a97042b7f1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.291767 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d149e0b2a5ab44a2ba2f81157a507967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.293280 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:53.293766 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d149e0b2a5ab44a2ba2f81157a507967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.296142 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed2604a090c94c78bd4f7a1dea6f81b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.298339 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d7b779d724df4863bb13a5140dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.300051 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c707458d1c094d87b2204683b28b49e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.304181 5837 rpcz_store.cc:276] 0507 08:11:51.468816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.511343 (+ 42527us) service_pool.cc:225] Handling call
0507 08:11:53.212972 (+1701629us) ts_tablet_manager.cc:1120] Deleting tablet 98880a5ff9604cb4981c033c99e80be7
0507 08:11:53.212981 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.230631 (+ 17650us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.238133 (+ 7502us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.251300 (+ 13167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701413}
I20240507 08:11:53.304570 5837 tablet_replica.cc:330] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.304801 5837 raft_consensus.cc:2229] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.304816 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 98880a5ff9604cb4981c033c99e80be7 (table default.Table_11f043808d86473e9d1ff0c532d9c4c0 [id=245f6e995eca47f38bdc7495b40a6299]) successfully deleted
I20240507 08:11:53.305565 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.306874 5557 tablet_service.cc:1467] Processing CreateTablet for tablet c707458d1c094d87b2204683b28b49e4 (DEFAULT_TABLE table=default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.310444 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8d15746afd6042b29e476bb2ecfa6c6e" } new_table_name: "default.Table_6d066df70e044e5d8fab4f4806216b98" modify_external_catalogs: true
I20240507 08:11:53.310734 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e447562e82347a584e39586500c27c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.312304 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ced879f35e14106b2609c78b7f24002. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.313035 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4504b1ed3d449619926fe80469d9790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.314260 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7947ca2d8324dbb9c909ec50852fed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.316819 5841 ts_tablet_manager.cc:1888] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.317260 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.318094 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67c8624285d946e597227430a5714e2f consensus state has no leader
I20240507 08:11:53.321734 5839 tablet_bootstrap.cc:492] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:53.321916 5839 ts_tablet_manager.cc:1367] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.491s user 0.012s sys 0.031s
I20240507 08:11:53.324641 5841 log.cc:1196] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c68b6c08f00d46779a31f4e9eed99f9f
W20240507 08:11:53.331214 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7027b4cefa4d41659f819ee4326470c4 consensus state has no leader
I20240507 08:11:53.331764 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c1becaf1c2694657b87263a7e941b119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.332378 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1becaf1c2694657b87263a7e941b119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.334017 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 364903504aa14d08b050a0cf83b82c71 consensus state has no leader
I20240507 08:11:53.334403 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_156b8cd7adba496a83c3269a663133f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.334985 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_156b8cd7adba496a83c3269a663133f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.306803 6024 log.cc:924] Time spent T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.134s user 0.000s sys 0.000s
W20240507 08:11:53.336472 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 070fdbf0b00b4582a184729c2f2a4b69 consensus state has no leader
I20240507 08:11:53.336557 5841 ts_tablet_manager.cc:1909] T c68b6c08f00d46779a31f4e9eed99f9f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:53.336834 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5aa2ec19328a431f8277539512fac60e" } modify_external_catalogs: true
I20240507 08:11:53.337028 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5aa2ec19328a431f8277539512fac60e" } modify_external_catalogs: true
I20240507 08:11:53.337411 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 105) took 2269 ms. Trace:
I20240507 08:11:53.337504 5841 rpcz_store.cc:276] 0507 08:11:51.067052 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.067111 (+ 59us) service_pool.cc:225] Handling call
0507 08:11:52.608931 (+1541820us) ts_tablet_manager.cc:1120] Deleting tablet c68b6c08f00d46779a31f4e9eed99f9f
0507 08:11:52.608942 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.205251 (+596309us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.316709 (+111458us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.336885 (+ 20176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541563}
I20240507 08:11:53.338323 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c68b6c08f00d46779a31f4e9eed99f9f (table default.Table_8da8971bb4e54275af1bd500580ef3c4 [id=73ef2c7a9e43402f906d455ab1006e72]) successfully deleted
I20240507 08:11:53.339176 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0c6246ffb46749509ee35f618aacb3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.339722 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c6246ffb46749509ee35f618aacb3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.312858 6008 raft_consensus.cc:482] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:53.340415 6008 raft_consensus.cc:504] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.341117 6008 leader_election.cc:288] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:53.333853 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 with delete_type TABLET_DATA_DELETED (replaced by tablet f49dd4772f8e46478638b29c009c83ea) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.343744 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb78ad08a4e759eb9344cbabb6a9d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:53.344024 5444 raft_consensus.cc:2454] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:53.345213 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 168e49ae24314a7aae2291485ddeda44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.346133 5656 leader_election.cc:302] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:53.346257 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b47f75826f1a465a9c38010bdf4d65ce" } modify_external_catalogs: true
I20240507 08:11:53.346452 6008 raft_consensus.cc:2781] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:53.346455 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b47f75826f1a465a9c38010bdf4d65ce" } modify_external_catalogs: true
I20240507 08:11:53.346561 6008 raft_consensus.cc:482] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:53.346663 6008 raft_consensus.cc:3037] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:53.347069 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_efefbeaaab494b34a0acd79bec1f4185"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.347608 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efefbeaaab494b34a0acd79bec1f4185 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.349550 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb78ad08a4e759eb9344cbabb6a9d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:53.350241 5654 leader_election.cc:341] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.352243 6008 raft_consensus.cc:504] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.353778 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb78ad08a4e759eb9344cbabb6a9d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:53.354041 5444 raft_consensus.cc:3037] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:53.354782 6008 leader_election.cc:288] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:53.357548 5840 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.357724 5840 ts_tablet_manager.cc:1367] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.006s
I20240507 08:11:53.358517 5840 raft_consensus.cc:348] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.358711 5840 raft_consensus.cc:374] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.358795 5840 raft_consensus.cc:729] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.359028 5840 consensus_queue.cc:260] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.360172 5840 ts_tablet_manager.cc:1398] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:53.362185 5839 raft_consensus.cc:348] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:53.362870 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 106) took 2248 ms. Trace:
I20240507 08:11:53.368234 5444 raft_consensus.cc:2454] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:53.368758 5656 leader_election.cc:302] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:53.370371 6008 raft_consensus.cc:2781] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:53.370790 6008 raft_consensus.cc:686] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:53.371353 6008 consensus_queue.cc:237] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.372522 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.372923 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16a4234bbc8e4c278d8ff310e4047b33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.369283 5841 rpcz_store.cc:276] 0507 08:11:51.089348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.089399 (+ 51us) service_pool.cc:225] Handling call
0507 08:11:53.337791 (+2248392us) ts_tablet_manager.cc:1120] Deleting tablet 5cab176c4bd34f38b14ac096f79b912e
0507 08:11:53.337801 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.337937 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2248164}
W20240507 08:11:53.373531 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16a4234bbc8e4c278d8ff310e4047b33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.373562 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5cab176c4bd34f38b14ac096f79b912e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5cab176c4bd34f38b14ac096f79b912e already in progress: creating tablet
I20240507 08:11:53.381656 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c58677b45a2b461490f3de5ca89c7b21" } modify_external_catalogs: true
I20240507 08:11:53.381887 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c58677b45a2b461490f3de5ca89c7b21" } modify_external_catalogs: true
I20240507 08:11:53.385016 5841 tablet_replica.cc:330] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:53.385231 5841 raft_consensus.cc:2229] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.386153 5838 ts_tablet_manager.cc:1888] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.386299 5841 raft_consensus.cc:2258] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:53.387652 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2de528d9fc046f183f70388cffab6c8 consensus state has no leader
I20240507 08:11:53.388856 5841 ts_tablet_manager.cc:1875] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.386495 5839 raft_consensus.cc:374] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.390211 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ca27a6c33fe74eff87c2e753756029f9" } modify_external_catalogs: true
I20240507 08:11:53.390395 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ca27a6c33fe74eff87c2e753756029f9" } modify_external_catalogs: true
I20240507 08:11:53.394294 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87377661d9374cef8ab1a97042b7f1b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.394696 5838 log.cc:1196] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/cbf73c218b944679a12d769b1929e535
I20240507 08:11:53.397498 5839 raft_consensus.cc:729] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.305068 5837 pending_rounds.cc:62] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:53.399329 5837 pending_rounds.cc:68] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924725104377856 op_type: NO_OP noop_request { }
I20240507 08:11:53.399541 5837 raft_consensus.cc:2866] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:53.399709 5837 raft_consensus.cc:2258] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.400938 5837 ts_tablet_manager.cc:1875] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.410116 5837 ts_tablet_manager.cc:1888] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.410264 5837 log.cc:1196] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/87e1f43426b14f98b985656f8d0d7141
I20240507 08:11:53.410720 5837 ts_tablet_manager.cc:1909] T 87e1f43426b14f98b985656f8d0d7141 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.411695 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 134) took 1773 ms. Trace:
I20240507 08:11:53.411826 5837 rpcz_store.cc:276] 0507 08:11:51.638024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.638065 (+ 41us) service_pool.cc:225] Handling call
0507 08:11:53.304504 (+1666439us) ts_tablet_manager.cc:1120] Deleting tablet 87e1f43426b14f98b985656f8d0d7141
0507 08:11:53.304513 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.406289 (+101776us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.409845 (+ 3556us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.411631 (+ 1786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666054}
I20240507 08:11:53.412439 5837 tablet_replica.cc:330] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.412796 5837 raft_consensus.cc:2229] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.413082 5837 raft_consensus.cc:2258] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.414222 5837 ts_tablet_manager.cc:1875] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.414258 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 87e1f43426b14f98b985656f8d0d7141 (table default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]) successfully deleted
I20240507 08:11:53.426352 5837 ts_tablet_manager.cc:1888] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.427214 5837 log.cc:1196] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/708d4bee30cc45bd9b86c56beb2c3ece
I20240507 08:11:53.427996 5837 ts_tablet_manager.cc:1909] T 708d4bee30cc45bd9b86c56beb2c3ece P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.429639 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 135) took 1791 ms. Trace:
I20240507 08:11:53.429832 5837 rpcz_store.cc:276] 0507 08:11:51.638153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.638218 (+ 65us) service_pool.cc:225] Handling call
0507 08:11:53.412276 (+1774058us) ts_tablet_manager.cc:1120] Deleting tablet 708d4bee30cc45bd9b86c56beb2c3ece
0507 08:11:53.412317 (+ 41us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.420092 (+ 7775us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.426145 (+ 6053us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.429489 (+ 3344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773772}
I20240507 08:11:53.431103 5837 tablet_replica.cc:330] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.431437 5837 raft_consensus.cc:2229] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.432078 5837 raft_consensus.cc:2258] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:53.432498 5995 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.532s user 0.004s sys 0.023s
W20240507 08:11:53.432623 5995 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.536s user 0.004s sys 0.023s
I20240507 08:11:53.433480 5837 ts_tablet_manager.cc:1875] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.437927 5840 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.438829 5843 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:53.439028 5843 ts_tablet_manager.cc:1367] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.459s user 0.004s sys 0.008s
I20240507 08:11:53.439837 5843 raft_consensus.cc:348] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.440093 5843 raft_consensus.cc:374] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.440222 5843 raft_consensus.cc:729] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.440461 5843 consensus_queue.cc:260] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.441056 5843 ts_tablet_manager.cc:1398] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:53.441371 5843 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:53.442360 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (DEFAULT_TABLE table=default.Table_7f36bf46aad74e46a282d6d05585b60d [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.442777 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.443185 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (DEFAULT_TABLE table=default.Table_7f36bf46aad74e46a282d6d05585b60d [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.443532 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.444409 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 835d7b779d724df4863bb13a5140dbf0 (DEFAULT_TABLE table=default.Table_1ccda05e59e0486b8a26054c5514666f [id=83f625713cda401f90e513ddef199bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.447932 5563 tablet_service.cc:1467] Processing CreateTablet for tablet e555a89d340448bc852f2441ca850213 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.449242 5837 ts_tablet_manager.cc:1888] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.449509 5837 log.cc:1196] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/400d2a7b0e224e46a5a580ca22e96eab
I20240507 08:11:53.450330 5837 ts_tablet_manager.cc:1909] T 400d2a7b0e224e46a5a580ca22e96eab P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.451892 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 132) took 1814 ms. Trace:
I20240507 08:11:53.452108 5837 rpcz_store.cc:276] 0507 08:11:51.637743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.637793 (+ 50us) service_pool.cc:225] Handling call
0507 08:11:53.430970 (+1793177us) ts_tablet_manager.cc:1120] Deleting tablet 400d2a7b0e224e46a5a580ca22e96eab
0507 08:11:53.430979 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.441667 (+ 10688us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.447862 (+ 6195us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.451822 (+ 3960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792329}
I20240507 08:11:53.452853 5837 tablet_replica.cc:330] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.453207 5837 raft_consensus.cc:2229] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.453684 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 400d2a7b0e224e46a5a580ca22e96eab (table default.Table_8e33d7020c354b828f678dbbd57d5832 [id=079eefc78b9445f098838f484af7a3ad]) successfully deleted
I20240507 08:11:53.453768 5837 pending_rounds.cc:62] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:53.453938 5837 pending_rounds.cc:68] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924724601184256 op_type: NO_OP noop_request { }
I20240507 08:11:53.454166 5837 raft_consensus.cc:2866] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:53.454370 5837 raft_consensus.cc:2258] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.455747 5837 ts_tablet_manager.cc:1875] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.457651 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.458127 6008 consensus_queue.cc:1035] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:11:53.458235 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.463439 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97c978ec8bca4aa5b2ffaf22c3290747. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.464020 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 0118ef28b24641d68cf24ce9df966243 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.464301 5840 tablet_bootstrap.cc:654] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.464730 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118ef28b24641d68cf24ce9df966243. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.467670 5837 ts_tablet_manager.cc:1888] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.467788 5837 log.cc:1196] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:53.468194 5837 ts_tablet_manager.cc:1909] T e1063aa5c7014d84b66f74ce28e3d890 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.469521 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 0118ef28b24641d68cf24ce9df966243 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.469911 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118ef28b24641d68cf24ce9df966243. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.474737 5249 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 7027b4cefa4d41659f819ee4326470c4 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7027b4cefa4d41659f819ee4326470c4 consensus state has no leader
W20240507 08:11:53.475167 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 395669us
I20240507 08:11:53.476037 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34e4e33f52df432b8d043171d7232c2a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:53.476271 5718 raft_consensus.cc:3037] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:53.479475 5718 raft_consensus.cc:2454] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:53.468987 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 133) took 1831 ms. Trace:
I20240507 08:11:53.483979 5837 rpcz_store.cc:276] 0507 08:11:51.637864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.638645 (+ 781us) service_pool.cc:225] Handling call
0507 08:11:53.452727 (+1814082us) ts_tablet_manager.cc:1120] Deleting tablet e1063aa5c7014d84b66f74ce28e3d890
0507 08:11:53.452739 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.462861 (+ 10122us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.467543 (+ 4682us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.468951 (+ 1408us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1813834}
W20240507 08:11:53.484508 5377 consensus_peers.cc:485] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98880a5ff9604cb4981c033c99e80be7. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:53.484927 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e1063aa5c7014d84b66f74ce28e3d890 (table default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]) successfully deleted
I20240507 08:11:53.485455 5837 tablet_replica.cc:330] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.485762 5837 raft_consensus.cc:2229] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.486084 5377 leader_election.cc:302] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:53.486555 5837 raft_consensus.cc:2258] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.487118 5994 raft_consensus.cc:2781] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:53.488739 5994 raft_consensus.cc:686] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:53.489118 5994 consensus_queue.cc:237] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.495117 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 168e49ae24314a7aae2291485ddeda44
I20240507 08:11:53.496256 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.496366 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.498778 5837 ts_tablet_manager.cc:1875] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.504680 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 070fdbf0b00b4582a184729c2f2a4b69. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.505482 5843 tablet_bootstrap.cc:654] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.509387 5841 ts_tablet_manager.cc:1888] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.511906 5841 log.cc:1196] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/400d2a7b0e224e46a5a580ca22e96eab
I20240507 08:11:53.512477 5841 ts_tablet_manager.cc:1909] T 400d2a7b0e224e46a5a580ca22e96eab P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:53.513770 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 108) took 2421 ms. Trace:
I20240507 08:11:53.504755 5839 consensus_queue.cc:260] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:53.518051 5841 rpcz_store.cc:276] 0507 08:11:51.092558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.092936 (+ 378us) service_pool.cc:225] Handling call
0507 08:11:53.373241 (+2280305us) ts_tablet_manager.cc:1120] Deleting tablet 400d2a7b0e224e46a5a580ca22e96eab
0507 08:11:53.373251 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.415739 (+ 42488us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.509213 (+ 93474us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.513726 (+ 4513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2280102}
I20240507 08:11:53.518884 5843 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:53.520529 5843 ts_tablet_manager.cc:1367] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.000s
I20240507 08:11:53.520797 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 400d2a7b0e224e46a5a580ca22e96eab (table default.Table_8e33d7020c354b828f678dbbd57d5832 [id=079eefc78b9445f098838f484af7a3ad]) successfully deleted
I20240507 08:11:53.521332 5843 raft_consensus.cc:348] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.521589 5843 raft_consensus.cc:374] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.521920 5843 raft_consensus.cc:729] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.524436 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 812dd1ebd42f4e929838a0535c22ad69
I20240507 08:11:53.524345 5843 consensus_queue.cc:260] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.525240 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1b19d8e28f164c588946ea5e81ec320e
I20240507 08:11:53.525827 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.526757 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_004cc703a3f541a9a31c0f4338618023" } new_table_name: "default.Table_74a9e58e257645d6b826cab9e4134f48" modify_external_catalogs: true
I20240507 08:11:53.526852 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.527107 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.527483 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 81a1a42a22704974b1ef6cbd17d91854 (DEFAULT_TABLE table=default.Table_7959240cfedd4e58940d3093998d84c1 [id=9af183d91ab34faebbe34ea8c4ad4196]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.528532 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.529035 5843 ts_tablet_manager.cc:1398] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240507 08:11:53.531811 5843 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:53.532407 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.533089 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (DEFAULT_TABLE table=default.Table_7f36bf46aad74e46a282d6d05585b60d [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.535557 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 835d7b779d724df4863bb13a5140dbf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.536384 5838 ts_tablet_manager.cc:1909] T cbf73c218b944679a12d769b1929e535 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:53.536823 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 168) took 1192 ms. Trace:
I20240507 08:11:53.538081 5840 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.538268 5840 ts_tablet_manager.cc:1367] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.178s user 0.004s sys 0.001s
I20240507 08:11:53.539700 5840 raft_consensus.cc:348] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.539964 5840 raft_consensus.cc:374] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.540047 5840 raft_consensus.cc:729] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.540328 5840 consensus_queue.cc:260] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.540982 5840 ts_tablet_manager.cc:1398] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:53.536911 5843 tablet_bootstrap.cc:654] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.540606 5841 tablet_replica.cc:330] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:53.543737 5841 raft_consensus.cc:2229] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.544637 5841 raft_consensus.cc:2258] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.546566 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8e3ffaf1781e4b108f305346476905be
I20240507 08:11:53.546772 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0118ef28b24641d68cf24ce9df966243 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.547173 5839 ts_tablet_manager.cc:1398] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.210s user 0.003s sys 0.000s
I20240507 08:11:53.547155 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7f36bf46aad74e46a282d6d05585b60d" } new_table_name: "default.Table_101fc94f9f8b4a70886064fc2f62b7de" modify_external_catalogs: true
I20240507 08:11:53.547327 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c48da430b2554361ab7c682e78a6a54f" } new_table_name: "default.Table_616cd80c5f274e8ab4b39e0d82adf728" modify_external_catalogs: true
I20240507 08:11:53.547513 5839 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:53.547713 5841 ts_tablet_manager.cc:1875] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.548389 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.550145 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_aa4603034e7845b493166a810805b43c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.550494 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_64af22be42cd4e0d84bc79cfcdba4f20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.550660 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa4603034e7845b493166a810805b43c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.550935 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64af22be42cd4e0d84bc79cfcdba4f20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.551764 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a1a42a22704974b1ef6cbd17d91854. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.551935 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a58b7d46a51f41049815a957ddac3f55" } new_table_name: "default.Table_75283770e2c44851bcfa380f6385359d" modify_external_catalogs: true
I20240507 08:11:53.552175 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.556102 5841 ts_tablet_manager.cc:1888] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.560222 5840 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.561810 5840 tablet_bootstrap.cc:654] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.564842 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.564883 5837 ts_tablet_manager.cc:1888] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.565021 5837 log.cc:1196] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/ed0074cc70e248ac8b5e90cf57b441c3
I20240507 08:11:53.554193 5838 rpcz_store.cc:276] 0507 08:11:52.344290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.621505 (+277215us) service_pool.cc:225] Handling call
0507 08:11:52.622458 (+ 953us) ts_tablet_manager.cc:1120] Deleting tablet cbf73c218b944679a12d769b1929e535
0507 08:11:52.622468 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:52.740654 (+118186us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.184772 (+444118us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.536775 (+352003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":217}
I20240507 08:11:53.565548 5837 ts_tablet_manager.cc:1909] T ed0074cc70e248ac8b5e90cf57b441c3 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.566329 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 136) took 1928 ms. Trace:
I20240507 08:11:53.566421 5837 rpcz_store.cc:276] 0507 08:11:51.638265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.639230 (+ 965us) service_pool.cc:225] Handling call
0507 08:11:53.485374 (+1846144us) ts_tablet_manager.cc:1120] Deleting tablet ed0074cc70e248ac8b5e90cf57b441c3
0507 08:11:53.485386 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.543064 (+ 57678us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.564767 (+ 21703us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.566280 (+ 1513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1845905}
I20240507 08:11:53.566833 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet ed0074cc70e248ac8b5e90cf57b441c3 (table default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]) successfully deleted
I20240507 08:11:53.568948 5841 log.cc:1196] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8ea5220905954dc9b078e00e0db69fd2
I20240507 08:11:53.569849 5841 ts_tablet_manager.cc:1909] T 8ea5220905954dc9b078e00e0db69fd2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:53.571758 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 107) took 2479 ms. Trace:
I20240507 08:11:53.573714 5841 rpcz_store.cc:276] 0507 08:11:51.092359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.115487 (+ 23128us) service_pool.cc:225] Handling call
0507 08:11:53.532603 (+2417116us) ts_tablet_manager.cc:1120] Deleting tablet 8ea5220905954dc9b078e00e0db69fd2
0507 08:11:53.532612 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.552767 (+ 20155us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.555992 (+ 3225us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.571702 (+ 15710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2416628}
I20240507 08:11:53.574652 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8ea5220905954dc9b078e00e0db69fd2 (table default.Table_b650283bfd9b448c8a0ba81f514f2831 [id=a2716884f32e46468e513117cd78b7af]) successfully deleted
I20240507 08:11:53.575011 5841 tablet_replica.cc:330] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:53.576884 5841 raft_consensus.cc:2229] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.577486 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c96ab339396440379b168db4805f8779"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.578032 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c96ab339396440379b168db4805f8779 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.578100 5841 raft_consensus.cc:2258] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.578949 5841 ts_tablet_manager.cc:1875] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.579054 5840 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.579272 5840 ts_tablet_manager.cc:1367] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240507 08:11:53.579413 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dfc198b816c048acb4142d4ede67abb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.579885 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfc198b816c048acb4142d4ede67abb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.580185 5840 raft_consensus.cc:348] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.580399 5840 raft_consensus.cc:374] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.580484 5840 raft_consensus.cc:729] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.580729 5840 consensus_queue.cc:260] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.581488 5840 ts_tablet_manager.cc:1398] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:53.581952 5840 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.591713 5837 tablet_replica.cc:330] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.591974 5837 raft_consensus.cc:2229] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.592394 5837 raft_consensus.cc:2258] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.592813 5840 tablet_bootstrap.cc:654] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.598310 5837 ts_tablet_manager.cc:1875] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.614996 5840 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.615254 5840 ts_tablet_manager.cc:1367] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20240507 08:11:53.615581 5837 ts_tablet_manager.cc:1888] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.615705 5837 log.cc:1196] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8ea5220905954dc9b078e00e0db69fd2
I20240507 08:11:53.616158 5837 ts_tablet_manager.cc:1909] T 8ea5220905954dc9b078e00e0db69fd2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.616802 5840 raft_consensus.cc:348] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.617098 5840 raft_consensus.cc:374] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.617558 5840 raft_consensus.cc:729] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.617914 5840 consensus_queue.cc:260] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.619676 5840 ts_tablet_manager.cc:1398] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:11:53.620144 5840 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.622059 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 137) took 1983 ms. Trace:
I20240507 08:11:53.622196 5837 rpcz_store.cc:276] 0507 08:11:51.638404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.639497 (+ 1093us) service_pool.cc:225] Handling call
0507 08:11:53.591618 (+1952121us) ts_tablet_manager.cc:1120] Deleting tablet 8ea5220905954dc9b078e00e0db69fd2
0507 08:11:53.591628 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.604782 (+ 13154us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.615451 (+ 10669us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.622021 (+ 6570us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5095,"tablet-delete.queue_time_us":1951951}
I20240507 08:11:53.622673 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8ea5220905954dc9b078e00e0db69fd2 (table default.Table_b650283bfd9b448c8a0ba81f514f2831 [id=a2716884f32e46468e513117cd78b7af]) successfully deleted
I20240507 08:11:53.623150 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 138) took 1982 ms. Trace:
I20240507 08:11:53.623270 5837 rpcz_store.cc:276] 0507 08:11:51.641069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.641459 (+ 390us) service_pool.cc:225] Handling call
0507 08:11:53.622943 (+1981484us) ts_tablet_manager.cc:1120] Deleting tablet 344bf4ab39f74accb383d695578de604
0507 08:11:53.622988 (+ 45us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.623111 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1981229}
I20240507 08:11:53.623567 5837 tablet_replica.cc:330] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.623785 5837 raft_consensus.cc:2229] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.623952 5837 raft_consensus.cc:2258] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:53.624133 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 344bf4ab39f74accb383d695578de604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344bf4ab39f74accb383d695578de604 already in progress: creating tablet
I20240507 08:11:53.624501 5837 ts_tablet_manager.cc:1875] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.625275 5840 tablet_bootstrap.cc:654] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.633312 5837 ts_tablet_manager.cc:1888] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.633483 5837 log.cc:1196] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/44f6f62c56cb4c29bf05914ca461a559
I20240507 08:11:53.633831 5837 ts_tablet_manager.cc:1909] T 44f6f62c56cb4c29bf05914ca461a559 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.634611 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 140) took 1984 ms. Trace:
I20240507 08:11:53.634716 5837 rpcz_store.cc:276] 0507 08:11:51.649642 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.649762 (+ 120us) service_pool.cc:225] Handling call
0507 08:11:53.623510 (+1973748us) ts_tablet_manager.cc:1120] Deleting tablet 44f6f62c56cb4c29bf05914ca461a559
0507 08:11:53.623518 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.627845 (+ 4327us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.633204 (+ 5359us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.634562 (+ 1358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1972093}
I20240507 08:11:53.635102 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 44f6f62c56cb4c29bf05914ca461a559 (table default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]) successfully deleted
I20240507 08:11:53.637547 5837 tablet_replica.cc:330] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.637763 5837 raft_consensus.cc:2229] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.637957 5837 raft_consensus.cc:2258] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.639315 5841 ts_tablet_manager.cc:1888] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.640944 5837 ts_tablet_manager.cc:1875] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.642552 5841 log.cc:1196] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:53.648259 5841 ts_tablet_manager.cc:1909] T e1063aa5c7014d84b66f74ce28e3d890 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:53.648617 5837 ts_tablet_manager.cc:1888] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.648779 5837 log.cc:1196] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/87ede5b03251463d8608d15f037e4484
I20240507 08:11:53.649222 5837 ts_tablet_manager.cc:1909] T 87ede5b03251463d8608d15f037e4484 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.649425 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 110) took 2520 ms. Trace:
I20240507 08:11:53.649610 5841 rpcz_store.cc:276] 0507 08:11:51.129055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.131987 (+ 2932us) service_pool.cc:225] Handling call
0507 08:11:53.574922 (+2442935us) ts_tablet_manager.cc:1120] Deleting tablet e1063aa5c7014d84b66f74ce28e3d890
0507 08:11:53.574932 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.613510 (+ 38578us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.639182 (+ 25672us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.649386 (+ 10204us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2442716}
I20240507 08:11:53.650121 5841 tablet_replica.cc:330] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:53.650219 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e1063aa5c7014d84b66f74ce28e3d890 (table default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]) successfully deleted
I20240507 08:11:53.650035 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 139) took 2000 ms. Trace:
I20240507 08:11:53.650516 5837 rpcz_store.cc:276] 0507 08:11:51.649506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.649906 (+ 400us) service_pool.cc:225] Handling call
0507 08:11:53.637472 (+1987566us) ts_tablet_manager.cc:1120] Deleting tablet 87ede5b03251463d8608d15f037e4484
0507 08:11:53.637481 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.644264 (+ 6783us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.648497 (+ 4233us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.649999 (+ 1502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1984148}
I20240507 08:11:53.651804 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 87ede5b03251463d8608d15f037e4484 (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:11:53.652406 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 141) took 1960 ms. Trace:
I20240507 08:11:53.652559 5837 rpcz_store.cc:276] 0507 08:11:51.692334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.692389 (+ 55us) service_pool.cc:225] Handling call
0507 08:11:53.652257 (+1959868us) ts_tablet_manager.cc:1120] Deleting tablet d9ff5e1b6a9a476c8295a0d0ee85aecc
0507 08:11:53.652266 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.652363 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1949802}
W20240507 08:11:53.653142 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ff5e1b6a9a476c8295a0d0ee85aecc already in progress: creating tablet
I20240507 08:11:53.653558 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 143) took 1950 ms. Trace:
I20240507 08:11:53.653761 5837 rpcz_store.cc:276] 0507 08:11:51.702842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.702944 (+ 102us) service_pool.cc:225] Handling call
0507 08:11:53.653432 (+1950488us) ts_tablet_manager.cc:1120] Deleting tablet f922a7519084409db98693ee8797592b
0507 08:11:53.653441 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.653518 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1949471}
W20240507 08:11:53.654261 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f922a7519084409db98693ee8797592b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f922a7519084409db98693ee8797592b already in progress: creating tablet
I20240507 08:11:53.654650 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 146) took 1943 ms. Trace:
I20240507 08:11:53.654845 5837 rpcz_store.cc:276] 0507 08:11:51.711048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.711089 (+ 41us) service_pool.cc:225] Handling call
0507 08:11:53.654523 (+1943434us) ts_tablet_manager.cc:1120] Deleting tablet ebc6cc72bb7e4721a8ab559102273db5
0507 08:11:53.654531 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.654622 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943261}
W20240507 08:11:53.655277 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ebc6cc72bb7e4721a8ab559102273db5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc6cc72bb7e4721a8ab559102273db5 already in progress: creating tablet
I20240507 08:11:53.655612 5837 tablet_replica.cc:330] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.655844 5837 raft_consensus.cc:2229] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.656188 5837 raft_consensus.cc:2258] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.656776 5837 ts_tablet_manager.cc:1875] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:53.651464 5377 consensus_peers.cc:485] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87ede5b03251463d8608d15f037e4484. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:53.658077 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.658226 5841 raft_consensus.cc:2229] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.658664 5841 pending_rounds.cc:62] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:53.658751 5841 pending_rounds.cc:68] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7024924718281695232 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "34ffe98d424a4eb5a08be067f96a50f9" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 4 new_table_name: "default.Table_65d6fd82bd8f4528ad808fb91dd27790" dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" new_extra_config { } }
I20240507 08:11:53.661098 5907 raft_consensus.cc:482] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:53.661283 5907 raft_consensus.cc:504] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.662039 5907 leader_election.cc:288] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:53.656014 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 170) took 1137 ms. Trace:
I20240507 08:11:53.662685 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:53.662902 5443 raft_consensus.cc:2454] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:53.663103 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.663328 5656 leader_election.cc:302] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:53.663671 5907 raft_consensus.cc:2781] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:53.663785 5907 raft_consensus.cc:482] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:53.663909 5907 raft_consensus.cc:3037] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:53.664033 5837 ts_tablet_manager.cc:1888] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.664163 5837 log.cc:1196] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c9b8044e4a0a41218f67513b14c0ebeb
I20240507 08:11:53.664536 5837 ts_tablet_manager.cc:1909] T c9b8044e4a0a41218f67513b14c0ebeb P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.665390 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 156) took 1813 ms. Trace:
I20240507 08:11:53.665491 5837 rpcz_store.cc:276] 0507 08:11:51.852251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.854191 (+ 1940us) service_pool.cc:225] Handling call
0507 08:11:53.655554 (+1801363us) ts_tablet_manager.cc:1120] Deleting tablet c9b8044e4a0a41218f67513b14c0ebeb
0507 08:11:53.655562 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.660522 (+ 4960us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.663926 (+ 3404us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.665287 (+ 1361us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1792335}
I20240507 08:11:53.665964 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c9b8044e4a0a41218f67513b14c0ebeb (table default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]) successfully deleted
I20240507 08:11:53.666429 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 171) took 1775 ms. Trace:
I20240507 08:11:53.666642 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e555a89d340448bc852f2441ca850213. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.666553 5837 rpcz_store.cc:276] 0507 08:11:51.890601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.065029 (+174428us) service_pool.cc:225] Handling call
0507 08:11:53.666253 (+1601224us) ts_tablet_manager.cc:1120] Deleting tablet e49dfd218a24464fbac39a3f4b477462
0507 08:11:53.666263 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.666368 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1600420}
I20240507 08:11:53.666682 5907 raft_consensus.cc:504] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:53.666970 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e49dfd218a24464fbac39a3f4b477462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e49dfd218a24464fbac39a3f4b477462 already in progress: creating tablet
I20240507 08:11:53.667526 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 173) took 1776 ms. Trace:
I20240507 08:11:53.667610 5837 rpcz_store.cc:276] 0507 08:11:51.890742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.069203 (+178461us) service_pool.cc:225] Handling call
0507 08:11:53.667391 (+1598188us) ts_tablet_manager.cc:1120] Deleting tablet 5228590c8e7d4c34b3acab9eb16665d3
0507 08:11:53.667400 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.667479 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597801}
I20240507 08:11:53.667742 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:53.667943 5443 raft_consensus.cc:3037] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:53.668098 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5228590c8e7d4c34b3acab9eb16665d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5228590c8e7d4c34b3acab9eb16665d3 already in progress: creating tablet
I20240507 08:11:53.668478 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 180) took 1676 ms. Trace:
I20240507 08:11:53.668569 5837 rpcz_store.cc:276] 0507 08:11:51.991686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.347196 (+355510us) service_pool.cc:225] Handling call
0507 08:11:53.668326 (+1321130us) ts_tablet_manager.cc:1120] Deleting tablet 60bc7bc06e494adcb6a8641b50b90e6c
0507 08:11:53.668346 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.668431 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298969}
W20240507 08:11:53.668922 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 60bc7bc06e494adcb6a8641b50b90e6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60bc7bc06e494adcb6a8641b50b90e6c already in progress: creating tablet
I20240507 08:11:53.669302 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 183) took 1635 ms. Trace:
I20240507 08:11:53.669401 5837 rpcz_store.cc:276] 0507 08:11:52.033681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.378605 (+344924us) service_pool.cc:225] Handling call
0507 08:11:53.669169 (+1290564us) ts_tablet_manager.cc:1120] Deleting tablet 270a1fd15925416483c211021549315c
0507 08:11:53.669178 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.669260 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1286694}
W20240507 08:11:53.669795 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 270a1fd15925416483c211021549315c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 270a1fd15925416483c211021549315c already in progress: creating tablet
I20240507 08:11:53.670089 5837 tablet_replica.cc:330] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.670276 5837 raft_consensus.cc:2229] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.670645 5443 raft_consensus.cc:2454] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:53.670697 5837 raft_consensus.cc:2258] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.671082 5907 leader_election.cc:288] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:53.671054 5656 leader_election.cc:302] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:53.671514 6008 raft_consensus.cc:2781] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:53.671694 6008 raft_consensus.cc:686] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:53.671710 5837 ts_tablet_manager.cc:1875] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.671965 6008 consensus_queue.cc:237] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:53.672363 5841 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:35094: Aborted: Op aborted
I20240507 08:11:53.672768 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:35094 (request call id 112) took 2491 ms. Trace:
I20240507 08:11:53.664785 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09adb78ad08a4e759eb9344cbabb6a9d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:53.673128 5841 rpcz_store.cc:276] 0507 08:11:51.180999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.182496 (+ 1497us) service_pool.cc:225] Handling call
0507 08:11:53.672717 (+2490221us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:11:51.188294 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0507 08:11:51.188395 (+ 101us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0507 08:11:51.299214 (+110819us) alter_schema_op.cc:59] Acquired schema lock
0507 08:11:51.299228 (+ 14us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0507 08:11:51.299276 (+ 48us) alter_schema_op.cc:119] START. Timestamp: 1715069511299242
0507 08:11:51.299280 (+ 4us) op_driver.cc:331] REPLICATION: starting
0507 08:11:51.299464 (+ 184us) log.cc:842] Serialized 173 byte log entry
0507 08:11:53.672055 (+2372591us) op_driver.cc:444] REPLICATION: finished
0507 08:11:53.672107 (+ 52us) op_driver.cc:372] HandleFailure(Aborted: Op aborted)
0507 08:11:53.672115 (+ 8us) alter_schema_op.cc:152] AlterSchemaCommitCallback: op aborted
Metrics: {"child_traces":[["op",{"prepare.queue_time_us":5386,"prepare.run_cpu_time_us":12014,"prepare.run_wall_time_us":111291,"raft.queue_time_us":488,"raft.run_cpu_time_us":22,"raft.run_wall_time_us":22,"replication_time_us":2359735}]]}
I20240507 08:11:53.673019 5582 raft_consensus.cc:3037] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:53.674631 5654 leader_election.cc:341] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.675228 5841 raft_consensus.cc:2258] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.679445 5838 rpcz_store.cc:276] 0507 08:11:52.438560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.692342 (+253782us) service_pool.cc:225] Handling call
0507 08:11:53.565432 (+873090us) ts_tablet_manager.cc:1120] Deleting tablet 5554ebc621c24d459668f2cefc20b641
0507 08:11:53.565447 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.575921 (+ 10474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":872840}
I20240507 08:11:53.680524 5837 ts_tablet_manager.cc:1888] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240507 08:11:53.680655 5837 log.cc:1196] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/cbf73c218b944679a12d769b1929e535
I20240507 08:11:53.680619 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:53.680572 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa932c6ad48f4ac183aabba8699cb148" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:53.680541 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.681106 5840 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
W20240507 08:11:53.681106 5654 leader_election.cc:341] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.681288 5840 ts_tablet_manager.cc:1367] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.061s user 0.001s sys 0.005s
I20240507 08:11:53.681507 5837 ts_tablet_manager.cc:1909] T cbf73c218b944679a12d769b1929e535 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.682155 5840 raft_consensus.cc:348] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.682296 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 184) took 1648 ms. Trace:
I20240507 08:11:53.682515 5840 raft_consensus.cc:374] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.682672 5840 raft_consensus.cc:729] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.682512 5837 rpcz_store.cc:276] 0507 08:11:52.033828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.382473 (+348645us) service_pool.cc:225] Handling call
0507 08:11:53.670016 (+1287543us) ts_tablet_manager.cc:1120] Deleting tablet cbf73c218b944679a12d769b1929e535
0507 08:11:53.670024 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.675792 (+ 5768us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.680217 (+ 4425us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.682257 (+ 2040us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275582}
I20240507 08:11:53.683080 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet cbf73c218b944679a12d769b1929e535 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
I20240507 08:11:53.683051 5840 consensus_queue.cc:260] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.683529 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 187) took 1647 ms. Trace:
I20240507 08:11:53.683655 5837 rpcz_store.cc:276] 0507 08:11:52.036557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.435414 (+398857us) service_pool.cc:225] Handling call
0507 08:11:53.683385 (+1247971us) ts_tablet_manager.cc:1120] Deleting tablet 5554ebc621c24d459668f2cefc20b641
0507 08:11:53.683395 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.683485 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050817}
W20240507 08:11:53.684091 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5554ebc621c24d459668f2cefc20b641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5554ebc621c24d459668f2cefc20b641 already in progress: creating tablet
I20240507 08:11:53.684473 5840 ts_tablet_manager.cc:1398] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:53.684674 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 194) took 1363 ms. Trace:
I20240507 08:11:53.684890 5837 rpcz_store.cc:276] 0507 08:11:52.321238 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.538171 (+216933us) service_pool.cc:225] Handling call
0507 08:11:53.684394 (+1146223us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:53.684437 (+ 43us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.684601 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051790}
W20240507 08:11:53.685473 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
I20240507 08:11:53.685721 5840 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.686106 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 193) took 1364 ms. Trace:
I20240507 08:11:53.686625 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.686470 5837 rpcz_store.cc:276] 0507 08:11:52.321159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.537983 (+216824us) service_pool.cc:225] Handling call
0507 08:11:53.685823 (+1147840us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:53.685835 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.686038 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053099}
I20240507 08:11:53.686961 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:53.687011 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:53.687613 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 195) took 1366 ms. Trace:
I20240507 08:11:53.687844 5837 rpcz_store.cc:276] 0507 08:11:52.321309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.538366 (+217057us) service_pool.cc:225] Handling call
0507 08:11:53.687288 (+1148922us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:53.687327 (+ 39us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.687546 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054231}
I20240507 08:11:53.688089 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0118ef28b24641d68cf24ce9df966243. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.688457 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:53.689018 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 190) took 1368 ms. Trace:
I20240507 08:11:53.689230 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89db2847c09b4c74b809bd13f78d0b8b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:53.689441 5654 leader_election.cc:341] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.689136 5837 rpcz_store.cc:276] 0507 08:11:52.320850 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.512741 (+191891us) service_pool.cc:225] Handling call
0507 08:11:53.688895 (+1176154us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:53.688903 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.688986 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054356}
I20240507 08:11:53.691000 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 172) took 1197 ms. Trace:
W20240507 08:11:53.691524 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
I20240507 08:11:53.691874 5837 tablet_replica.cc:330] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.692117 5837 raft_consensus.cc:2229] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.692351 5837 raft_consensus.cc:2258] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.692849 5838 rpcz_store.cc:276] 0507 08:11:52.489113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.699873 (+210760us) service_pool.cc:225] Handling call
0507 08:11:53.680316 (+980443us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:53.680326 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.686915 (+ 6589us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":979547}
I20240507 08:11:53.693018 5841 ts_tablet_manager.cc:1875] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.693001 5837 ts_tablet_manager.cc:1875] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.693279 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 173) took 1189 ms. Trace:
I20240507 08:11:53.693410 5838 rpcz_store.cc:276] 0507 08:11:52.504293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.700774 (+196481us) service_pool.cc:225] Handling call
0507 08:11:53.693140 (+992366us) ts_tablet_manager.cc:1120] Deleting tablet 09adb78ad08a4e759eb9344cbabb6a9d
0507 08:11:53.693148 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.693243 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992194}
I20240507 08:11:53.693969 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 174) took 1171 ms. Trace:
I20240507 08:11:53.694100 5838 rpcz_store.cc:276] 0507 08:11:52.522483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.700954 (+178471us) service_pool.cc:225] Handling call
0507 08:11:53.693837 (+992883us) ts_tablet_manager.cc:1120] Deleting tablet 5f2b43daab6b4deeac94fc6f1ff34a76
0507 08:11:53.693845 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.693934 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992696}
I20240507 08:11:53.694442 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 175) took 1170 ms. Trace:
I20240507 08:11:53.694604 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.694574 5838 rpcz_store.cc:276] 0507 08:11:52.529884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.701161 (+171277us) service_pool.cc:225] Handling call
0507 08:11:53.694310 (+993149us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:53.694318 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.694407 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":992994}
I20240507 08:11:53.695430 5840 tablet_bootstrap.cc:654] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.699257 6011 tablet.cc:1722] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:53.699292 5987 consensus_queue.cc:1035] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:53.700078 6012 tablet.cc:1722] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:53.701038 5837 ts_tablet_manager.cc:1888] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.701191 5837 log.cc:1196] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/34e4e33f52df432b8d043171d7232c2a
I20240507 08:11:53.703763 5974 raft_consensus.cc:482] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:53.705335 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c707458d1c094d87b2204683b28b49e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.706344 5837 ts_tablet_manager.cc:1909] T 34e4e33f52df432b8d043171d7232c2a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.707207 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 200) took 1385 ms. Trace:
I20240507 08:11:53.707322 5837 rpcz_store.cc:276] 0507 08:11:52.321872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.579115 (+257243us) service_pool.cc:225] Handling call
0507 08:11:53.691809 (+1112694us) ts_tablet_manager.cc:1120] Deleting tablet 34e4e33f52df432b8d043171d7232c2a
0507 08:11:53.691818 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.697137 (+ 5319us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.700922 (+ 3785us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.707149 (+ 6227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053521}
I20240507 08:11:53.707756 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 34e4e33f52df432b8d043171d7232c2a (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]) successfully deleted
I20240507 08:11:53.707808 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet cbf73c218b944679a12d769b1929e535 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
I20240507 08:11:53.708204 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 192) took 1387 ms. Trace:
W20240507 08:11:53.708304 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5554ebc621c24d459668f2cefc20b641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5554ebc621c24d459668f2cefc20b641 already in progress: creating tablet
I20240507 08:11:53.708302 5837 rpcz_store.cc:276] 0507 08:11:52.321032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.537751 (+216719us) service_pool.cc:225] Handling call
0507 08:11:53.708032 (+1170281us) ts_tablet_manager.cc:1120] Deleting tablet 5f2b43daab6b4deeac94fc6f1ff34a76
0507 08:11:53.708042 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.708151 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062278}
W20240507 08:11:53.708698 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
W20240507 08:11:53.708742 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2b43daab6b4deeac94fc6f1ff34a76 already in progress: creating tablet
I20240507 08:11:53.709003 5837 tablet_replica.cc:330] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.709211 5837 raft_consensus.cc:2229] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.709493 5837 pending_rounds.cc:62] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:53.709657 5837 pending_rounds.cc:68] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924726772531200 op_type: NO_OP noop_request { }
I20240507 08:11:53.709812 5837 raft_consensus.cc:2866] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:53.709926 5837 raft_consensus.cc:2258] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.710424 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 176) took 1159 ms. Trace:
I20240507 08:11:53.710781 5837 ts_tablet_manager.cc:1875] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.713349 5841 ts_tablet_manager.cc:1888] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:11:53.714864 5974 raft_consensus.cc:504] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:53.716816 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 09adb78ad08a4e759eb9344cbabb6a9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09adb78ad08a4e759eb9344cbabb6a9d already in progress: creating tablet
W20240507 08:11:53.717077 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2b43daab6b4deeac94fc6f1ff34a76 already in progress: creating tablet
I20240507 08:11:53.717120 5837 ts_tablet_manager.cc:1888] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:53.717258 5837 log.cc:1196] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/09adb78ad08a4e759eb9344cbabb6a9d
W20240507 08:11:53.717280 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:53.717691 5837 ts_tablet_manager.cc:1909] T 09adb78ad08a4e759eb9344cbabb6a9d P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.718474 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 191) took 1397 ms. Trace:
I20240507 08:11:53.718587 5837 rpcz_store.cc:276] 0507 08:11:52.320954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.537292 (+216338us) service_pool.cc:225] Handling call
0507 08:11:53.708902 (+1171610us) ts_tablet_manager.cc:1120] Deleting tablet 09adb78ad08a4e759eb9344cbabb6a9d
0507 08:11:53.708913 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.713866 (+ 4953us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.717027 (+ 3161us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.718404 (+ 1377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062222}
I20240507 08:11:53.719110 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 09adb78ad08a4e759eb9344cbabb6a9d (table default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]) successfully deleted
I20240507 08:11:53.719491 5837 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 201) took 1227 ms. Trace:
I20240507 08:11:53.719544 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:11:53.719579 5837 rpcz_store.cc:276] 0507 08:11:52.492321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.583137 (+ 90816us) service_pool.cc:225] Handling call
0507 08:11:53.719330 (+1136193us) ts_tablet_manager.cc:1120] Deleting tablet c33710c6b55744c4ba0843c228abded3
0507 08:11:53.719339 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.719439 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1056676}
W20240507 08:11:53.720005 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c33710c6b55744c4ba0843c228abded3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33710c6b55744c4ba0843c228abded3 already in progress: creating tablet
I20240507 08:11:53.718472 5840 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:53.720827 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.720921 5840 ts_tablet_manager.cc:1367] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240507 08:11:53.721413 5839 tablet_bootstrap.cc:654] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.721803 5840 raft_consensus.cc:348] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:53.722108 5840 raft_consensus.cc:374] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.722199 5840 raft_consensus.cc:729] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.722545 5837 tablet_replica.cc:330] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.722771 5837 raft_consensus.cc:2229] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:53.723085 5837 raft_consensus.cc:2258] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.723393 5840 consensus_queue.cc:260] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:53.724841 5837 ts_tablet_manager.cc:1875] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.726387 5840 ts_tablet_manager.cc:1398] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240507 08:11:53.726697 5840 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:53.732782 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.735770 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:53.736593 5840 tablet_bootstrap.cc:654] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.741250 5837 ts_tablet_manager.cc:1888] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:53.741385 5837 log.cc:1196] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/baa842552d00489998f83e68aa0d516b
I20240507 08:11:53.741255 5838 rpcz_store.cc:276] 0507 08:11:52.537895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.701326 (+163431us) service_pool.cc:225] Handling call
0507 08:11:53.696777 (+995451us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:53.696788 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.696861 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":995268}
I20240507 08:11:53.742132 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0c6246ffb46749509ee35f618aacb3ed" } new_table_name: "default.Table_342a9c864d774adf993c547b7e2f79d1" modify_external_catalogs: true
W20240507 08:11:53.742233 5517 leader_election.cc:341] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:11:53.742501 5837 ts_tablet_manager.cc:1909] T baa842552d00489998f83e68aa0d516b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:11:53.743407 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 907b5cb0e7a54903abe0b0d0e8d617dc consensus state has no leader
I20240507 08:11:53.743572 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet baa842552d00489998f83e68aa0d516b (table default.Table_5fc65aee40154e6296dc8b3f3076a4f3 [id=13e03112a7a641828153301cefc1774e]) successfully deleted
I20240507 08:11:53.743587 5974 leader_election.cc:288] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:53.743885 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b8acd47e87954187bc2d573c74b9290e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.744084 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
W20240507 08:11:53.744400 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8acd47e87954187bc2d573c74b9290e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.744442 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6bf34089e5ec419f97c0da9ec42d7eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf34089e5ec419f97c0da9ec42d7eec already in progress: creating tablet
W20240507 08:11:53.745256 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 191adef34efa4c3999d4642a80177294 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 191adef34efa4c3999d4642a80177294 already in progress: creating tablet
W20240507 08:11:53.745817 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 consensus state has no leader
I20240507 08:11:53.746166 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_32f389693bc94adbaae4903e3d430cc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.746641 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32f389693bc94adbaae4903e3d430cc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.747881 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet add94bcf0c844cb78059b7240c7e28c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet add94bcf0c844cb78059b7240c7e28c0 already in progress: creating tablet
I20240507 08:11:53.748248 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_105a7078d5c648b7ac16943b060dbcca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.748723 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_105a7078d5c648b7ac16943b060dbcca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.748960 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 80d82625304942d38b0261f95e235a41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80d82625304942d38b0261f95e235a41 already in progress: creating tablet
I20240507 08:11:53.750166 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c45ba94266f94dadbc071706db9038ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.750597 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c45ba94266f94dadbc071706db9038ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:53.751351 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d537bed29a2f47058255d2530b5c0d23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d537bed29a2f47058255d2530b5c0d23 already in progress: creating tablet
W20240507 08:11:53.751884 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1cd02a49359743c4b4e540ab0338d8ed consensus state has no leader
W20240507 08:11:53.752231 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bde62374dc7f4a67a071b6b658e501c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bde62374dc7f4a67a071b6b658e501c4 already in progress: creating tablet
I20240507 08:11:53.752691 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1b3804a3ddb546bdb6a9d9813e2372d0" } modify_external_catalogs: true
I20240507 08:11:53.752874 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1b3804a3ddb546bdb6a9d9813e2372d0" } modify_external_catalogs: true
W20240507 08:11:53.753408 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet cb7acbdb357445adb9ce69103c782a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7acbdb357445adb9ce69103c782a7f already in progress: creating tablet
W20240507 08:11:53.754176 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d10b38120dfe4b9ab3c39cf584e4ff13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d10b38120dfe4b9ab3c39cf584e4ff13 already in progress: creating tablet
W20240507 08:11:53.754933 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 193ec0b463d04835b74df80229c85cde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 193ec0b463d04835b74df80229c85cde already in progress: creating tablet
W20240507 08:11:53.755620 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 107d9b3f95e14044854ad0bf91a268ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107d9b3f95e14044854ad0bf91a268ee already in progress: creating tablet
I20240507 08:11:53.756075 5837 tablet_replica.cc:330] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.756268 5837 raft_consensus.cc:2229] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.756420 5837 raft_consensus.cc:2258] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:53.757480 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 consensus state has no leader
I20240507 08:11:53.757833 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71f59cb7918c42d29419261d5f361830" } modify_external_catalogs: true
I20240507 08:11:53.757908 5837 ts_tablet_manager.cc:1875] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.758011 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71f59cb7918c42d29419261d5f361830" } modify_external_catalogs: true
I20240507 08:11:53.758919 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c3dd460f347c47e9990f8cd2f4aeae71" } modify_external_catalogs: true
I20240507 08:11:53.759115 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c3dd460f347c47e9990f8cd2f4aeae71" } modify_external_catalogs: true
I20240507 08:11:53.752236 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ede2eb6eb3b247c2ac451571f36e8e29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.762593 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ede2eb6eb3b247c2ac451571f36e8e29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.782617 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:53.783591 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_420e2e62b141454298f31e925f51f2d8" } modify_external_catalogs: true
I20240507 08:11:53.783931 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_420e2e62b141454298f31e925f51f2d8" } modify_external_catalogs: true
I20240507 08:11:53.784163 5837 ts_tablet_manager.cc:1888] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.784350 5837 log.cc:1196] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e1cf2df0d3bd46268e22713fcc135a74
I20240507 08:11:53.784734 5837 ts_tablet_manager.cc:1909] T e1cf2df0d3bd46268e22713fcc135a74 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.785705 5837 tablet_replica.cc:330] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.785786 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e1cf2df0d3bd46268e22713fcc135a74 (table default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]) successfully deleted
I20240507 08:11:53.785912 5837 raft_consensus.cc:2229] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.786103 5837 raft_consensus.cc:2258] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.786199 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.786661 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e63c3f3ecc84913ad05864be450e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.794895 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba277be774d32912d1174bec68b51 (DEFAULT_TABLE table=default.Table_e34782f4ad6e45c8a22146e8d9204e89 [id=ea7a6ae7098540f79eb1b001b73a41a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.795379 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba277be774d32912d1174bec68b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.800223 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 697611671ef64c53b0be4f5341b8f270
I20240507 08:11:53.800566 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 177) took 1226 ms. Trace:
I20240507 08:11:53.803148 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:53.803444 5255 catalog_manager.cc:5484] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:53.804059 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:53.807089 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e53409fe13e4a9585dbba648a25cc07 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:53.807474 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 697611671ef64c53b0be4f5341b8f270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.808732 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 88a2cfb5f046434c8340234a447a7e65 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:53.721801 5843 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:53.810550 5843 ts_tablet_manager.cc:1367] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.281s user 0.002s sys 0.033s
I20240507 08:11:53.810770 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8ed8bac124cb4de983f0fdbaf728975f
I20240507 08:11:53.811434 5843 raft_consensus.cc:348] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.811654 5843 raft_consensus.cc:374] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.811743 5843 raft_consensus.cc:729] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.811982 5843 consensus_queue.cc:260] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.812216 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.806644 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:53.812541 5843 ts_tablet_manager.cc:1398] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:53.812837 5843 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:53.726194 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.745698 5994 raft_consensus.cc:482] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:53.813732 5994 raft_consensus.cc:504] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.814177 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:53.814301 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ba2e8d1e01047fd9cc1c192e9628750 consensus state has no leader
I20240507 08:11:53.814424 5994 leader_election.cc:288] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:53.753405 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "baa842552d00489998f83e68aa0d516b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:53.756732 6033 raft_consensus.cc:482] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:53.814862 6033 raft_consensus.cc:504] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.815614 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.815652 6033 leader_election.cc:288] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:53.815843 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:53.764010 5378 consensus_peers.cc:485] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:53.816296 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 52336.8us
I20240507 08:11:53.782925 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.816823 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e63c3f3ecc84913ad05864be450e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.819967 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.819885 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f3ee2b04de4cdf8a621b39f1476577" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:53.821166 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b4504b1ed3d449619926fe80469d9790
W20240507 08:11:53.821872 5378 leader_election.cc:341] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.805445 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 697611671ef64c53b0be4f5341b8f270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.823623 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba277be774d32912d1174bec68b51 (DEFAULT_TABLE table=default.Table_e34782f4ad6e45c8a22146e8d9204e89 [id=ea7a6ae7098540f79eb1b001b73a41a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.824019 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba277be774d32912d1174bec68b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.824805 5520 leader_election.cc:302] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:53.829538 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.830511 5424 tablet_service.cc:1467] Processing CreateTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 (DEFAULT_TABLE table=default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.830883 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e58ba8c26d4f17af24eb379aca1f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.832592 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:53.834092 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d (DEFAULT_TABLE table=default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.834515 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a59c7257e74b4abaa4ea7a1ee9136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.835693 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.836088 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e63c3f3ecc84913ad05864be450e9b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.836930 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 697611671ef64c53b0be4f5341b8f270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.837106 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.837230 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1cf2df0d3bd46268e22713fcc135a74" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:53.837534 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.837850 5554 tablet_service.cc:1467] Processing CreateTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 (DEFAULT_TABLE table=default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:53.840240 5378 leader_election.cc:341] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.840258 5838 rpcz_store.cc:276] 0507 08:11:52.539633 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.701516 (+161883us) service_pool.cc:225] Handling call
0507 08:11:53.741526 (+1040010us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:53.741533 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.765599 (+ 24066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1017314}
I20240507 08:11:53.840863 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 67c8624285d946e597227430a5714e2f from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:53.841600 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba277be774d32912d1174bec68b51 (DEFAULT_TABLE table=default.Table_e34782f4ad6e45c8a22146e8d9204e89 [id=ea7a6ae7098540f79eb1b001b73a41a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.841789 5415 tablet_service.cc:1467] Processing CreateTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (DEFAULT_TABLE table=default.Table_16a4234bbc8e4c278d8ff310e4047b33 [id=799497aabd9b4f17bc42b655e947b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.841989 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 5228590c8e7d4c34b3acab9eb16665d3 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:53.842118 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7053cfaea4a54a402c9d826613c05 (DEFAULT_TABLE table=default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.842201 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f8f4205f1f48abaf4fcd7113ad44a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.845949 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.846401 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4b84ab3f274185a2b838ebb969ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.847306 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d (DEFAULT_TABLE table=default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.847326 5838 tablet_replica.cc:330] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:53.848922 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (DEFAULT_TABLE table=default.Table_16a4234bbc8e4c278d8ff310e4047b33 [id=799497aabd9b4f17bc42b655e947b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.849222 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e58ba8c26d4f17af24eb379aca1f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.850217 5837 ts_tablet_manager.cc:1875] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.852017 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.852303 5697 tablet_service.cc:1467] Processing CreateTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 (DEFAULT_TABLE table=default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.852707 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3e58ba8c26d4f17af24eb379aca1f60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.853120 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f8f4205f1f48abaf4fcd7113ad44a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.853778 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.855422 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4b84ab3f274185a2b838ebb969ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.856985 5839 tablet_bootstrap.cc:492] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:53.857712 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.857856 5837 ts_tablet_manager.cc:1888] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.857980 5837 log.cc:1196] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f968f122c39b4fd5b995e8a2848a52d4
I20240507 08:11:53.858003 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 with delete_type TABLET_DATA_DELETED (replaced by tablet 270a1fd15925416483c211021549315c) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.858388 5837 ts_tablet_manager.cc:1909] T f968f122c39b4fd5b995e8a2848a52d4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.858455 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7053cfaea4a54a402c9d826613c05 (DEFAULT_TABLE table=default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.858644 5838 raft_consensus.cc:2229] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.859494 5837 tablet_replica.cc:330] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.859462 5699 tablet_service.cc:1467] Processing CreateTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (DEFAULT_TABLE table=default.Table_16a4234bbc8e4c278d8ff310e4047b33 [id=799497aabd9b4f17bc42b655e947b3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.859694 5837 raft_consensus.cc:2229] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.859884 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f8f4205f1f48abaf4fcd7113ad44a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.859913 5837 raft_consensus.cc:2258] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.860718 5837 ts_tablet_manager.cc:1875] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.861950 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7053cfaea4a54a402c9d826613c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.866189 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d (DEFAULT_TABLE table=default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:53.866206 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:53.866595 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a59c7257e74b4abaa4ea7a1ee9136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.867693 5837 ts_tablet_manager.cc:1888] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.867866 5837 log.cc:1196] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/812dd1ebd42f4e929838a0535c22ad69
I20240507 08:11:53.868239 5837 ts_tablet_manager.cc:1909] T 812dd1ebd42f4e929838a0535c22ad69 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.869850 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.870097 5837 tablet_replica.cc:330] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.870285 5837 raft_consensus.cc:2229] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.870465 5837 raft_consensus.cc:2258] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.871210 5837 ts_tablet_manager.cc:1875] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.875277 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5e53409fe13e4a9585dbba648a25cc07
I20240507 08:11:53.875773 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 5e53409fe13e4a9585dbba648a25cc07 deleted and replaced by tablet b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:11:53.876159 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 88a2cfb5f046434c8340234a447a7e65
I20240507 08:11:53.876591 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 88a2cfb5f046434c8340234a447a7e65 deleted and replaced by tablet d451d03349794726a71c255a7f177436
I20240507 08:11:53.877038 5838 raft_consensus.cc:2258] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.879830 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7053cfaea4a54a402c9d826613c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.880215 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.880576 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a59c7257e74b4abaa4ea7a1ee9136d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.880640 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b4b84ab3f274185a2b838ebb969ab52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.881152 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.882731 5837 ts_tablet_manager.cc:1888] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.882928 5837 log.cc:1196] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/91c6cb7920df47aa97be6555c587798a
I20240507 08:11:53.883323 5837 ts_tablet_manager.cc:1909] T 91c6cb7920df47aa97be6555c587798a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.884287 5837 tablet_replica.cc:330] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.884565 5837 raft_consensus.cc:2229] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.884778 5837 raft_consensus.cc:2258] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.885679 5837 ts_tablet_manager.cc:1875] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.886168 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.886183 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d (DEFAULT_TABLE table=default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.886565 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5e3d23bbb6439a87b303dff1b44a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.886592 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.887075 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f3ee2b04de4cdf8a621b39f1476577" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:53.887365 5718 raft_consensus.cc:2454] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:53.887832 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f968f122c39b4fd5b995e8a2848a52d4 (table default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]) successfully deleted
W20240507 08:11:53.888111 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 168e49ae24314a7aae2291485ddeda44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168e49ae24314a7aae2291485ddeda44 already in progress: creating tablet
I20240507 08:11:53.888291 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 812dd1ebd42f4e929838a0535c22ad69 (table default.Table_f29e8299e1fc4d2ba2b6cd3036641274 [id=a764e2dd5a9841fe9ebc72c179b7858e]) successfully deleted
W20240507 08:11:53.888504 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1b19d8e28f164c588946ea5e81ec320e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b19d8e28f164c588946ea5e81ec320e already in progress: creating tablet
W20240507 08:11:53.888692 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8e3ffaf1781e4b108f305346476905be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e3ffaf1781e4b108f305346476905be already in progress: creating tablet
W20240507 08:11:53.888873 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b22334378eec4ab3885d0f9d2d812a3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22334378eec4ab3885d0f9d2d812a3e already in progress: creating tablet
W20240507 08:11:53.889032 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 697611671ef64c53b0be4f5341b8f270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 697611671ef64c53b0be4f5341b8f270 already in progress: creating tablet
W20240507 08:11:53.889196 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8ed8bac124cb4de983f0fdbaf728975f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed8bac124cb4de983f0fdbaf728975f already in progress: creating tablet
W20240507 08:11:53.889365 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4867e1cca1634b4ab6e36f9ed3204e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4867e1cca1634b4ab6e36f9ed3204e0a already in progress: creating tablet
I20240507 08:11:53.889624 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 91c6cb7920df47aa97be6555c587798a (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:11:53.890061 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 (DEFAULT_TABLE table=default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.890429 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba2e8d1e01047fd9cc1c192e9628750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.892477 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 67c8624285d946e597227430a5714e2f
I20240507 08:11:53.892932 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 67c8624285d946e597227430a5714e2f deleted and replaced by tablet 0d617213b959452ab6519809e56db128
I20240507 08:11:53.899930 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 400d2a7b0e224e46a5a580ca22e96eab from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:53.899910 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 4de7053cfaea4a54a402c9d826613c05 (DEFAULT_TABLE table=default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.900346 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4de7053cfaea4a54a402c9d826613c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.900874 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 (DEFAULT_TABLE table=default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.902040 5377 leader_election.cc:302] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:53.902348 6033 raft_consensus.cc:2781] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:53.902846 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 (DEFAULT_TABLE table=default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.903110 5839 ts_tablet_manager.cc:1367] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.310s user 0.007s sys 0.000s
I20240507 08:11:53.905282 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d (DEFAULT_TABLE table=default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.907840 6033 raft_consensus.cc:482] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:53.907984 6033 raft_consensus.cc:3037] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:53.908206 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.908398 5838 ts_tablet_manager.cc:1875] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.908596 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:53.910975 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d (DEFAULT_TABLE table=default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.911288 5994 raft_consensus.cc:482] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:11:53.911795 5377 leader_election.cc:341] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e1cf2df0d3bd46268e22713fcc135a74
I20240507 08:11:53.916268 5994 raft_consensus.cc:504] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:53.916989 5994 leader_election.cc:288] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:53.917129 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.917399 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.918440 5377 leader_election.cc:302] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:53.918998 5377 consensus_peers.cc:485] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 708d4bee30cc45bd9b86c56beb2c3ece. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:53.919143 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.919407 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba2e8d1e01047fd9cc1c192e9628750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.919469 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba277be774d32912d1174bec68b51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.920488 5994 raft_consensus.cc:2726] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:53.922726 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5e3d23bbb6439a87b303dff1b44a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.924680 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ba2e8d1e01047fd9cc1c192e9628750. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.927656 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_32f389693bc94adbaae4903e3d430cc6" } modify_external_catalogs: true
I20240507 08:11:53.927881 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_32f389693bc94adbaae4903e3d430cc6" } modify_external_catalogs: true
I20240507 08:11:53.928614 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 753019050fcf4dc3b69722c3eb475037 (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.929009 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753019050fcf4dc3b69722c3eb475037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.931267 5837 ts_tablet_manager.cc:1888] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.929744 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b5e3d23bbb6439a87b303dff1b44a1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.932983 5837 log.cc:1196] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:53.933460 5837 ts_tablet_manager.cc:1909] T 0a6a35991c18414b9bc714e9a0ba14d8 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.934056 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 (DEFAULT_TABLE table=default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.934419 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037c3c0af5e464bb23a311e552fcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.936028 5837 tablet_replica.cc:330] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:53.936296 5837 raft_consensus.cc:2229] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:53.936537 5837 raft_consensus.cc:2258] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:53.937445 5837 ts_tablet_manager.cc:1875] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:53.937837 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 753019050fcf4dc3b69722c3eb475037 (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.938269 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753019050fcf4dc3b69722c3eb475037. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.932477 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:53.930838 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 (DEFAULT_TABLE table=default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.942750 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 (DEFAULT_TABLE table=default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.944092 5837 ts_tablet_manager.cc:1888] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:53.944262 5837 log.cc:1196] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e0f3ee2b04de4cdf8a621b39f1476577
I20240507 08:11:53.944677 5837 ts_tablet_manager.cc:1909] T e0f3ee2b04de4cdf8a621b39f1476577 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:53.946326 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7c39ec9fec4331afabb39d1483ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.947701 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 (DEFAULT_TABLE table=default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.950510 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 (DEFAULT_TABLE table=default.Table_dfc198b816c048acb4142d4ede67abb7 [id=3b2ccfca7fae432bb6acfa7e375f6ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.951164 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d (DEFAULT_TABLE table=default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.951947 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7c39ec9fec4331afabb39d1483ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.946766 5839 raft_consensus.cc:348] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:53.953291 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 35e34b60b78b402faf51e003c97db787 (DEFAULT_TABLE table=default.Table_aa4603034e7845b493166a810805b43c [id=71fd821e13734824a24f76fa2c220f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.954414 5427 tablet_service.cc:1467] Processing CreateTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 (DEFAULT_TABLE table=default.Table_c45ba94266f94dadbc071706db9038ff [id=6f1cedff6040494bb0a9bfe571648812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.954770 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294baf04bf74dfe8500cb8c1452d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.955760 5843 tablet_bootstrap.cc:654] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:53.955799 5839 raft_consensus.cc:374] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:53.955919 5839 raft_consensus.cc:729] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:53.957412 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1c7c39ec9fec4331afabb39d1483ef57
W20240507 08:11:53.961050 5378 leader_election.cc:341] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:53.961820 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_431fe83bbf04464e84771e65abee1c42" } new_table_name: "default.Table_94b53721a4784e1b94415eb370b8dd9c" modify_external_catalogs: true
I20240507 08:11:53.963212 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_556bd9f671734de4a0546ca45974a3ba" } new_table_name: "default.Table_bef65763cc5e492f993abf19cd31e433" modify_external_catalogs: true
I20240507 08:11:53.964422 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_499cbcb48a784dcab2336eece710d4b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.964937 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_499cbcb48a784dcab2336eece710d4b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.966148 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4fb1a5ebaf7c4b01bd7123a48782485f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.966607 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fb1a5ebaf7c4b01bd7123a48782485f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.966711 5684 tablet_service.cc:1467] Processing CreateTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 (DEFAULT_TABLE table=default.Table_c45ba94266f94dadbc071706db9038ff [id=6f1cedff6040494bb0a9bfe571648812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.966837 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 (DEFAULT_TABLE table=default.Table_dfc198b816c048acb4142d4ede67abb7 [id=3b2ccfca7fae432bb6acfa7e375f6ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.967101 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294baf04bf74dfe8500cb8c1452d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.967388 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f22194aff40609a3cfa63b9b3aca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.967723 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_931a381db9684935b68533da48152c99" } new_table_name: "default.Table_81657f34e12b4ca0b2fd8ee26ab0e663" modify_external_catalogs: true
I20240507 08:11:53.968932 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2be2b9efedf84c4198bb1b2804f349d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.969347 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2be2b9efedf84c4198bb1b2804f349d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.972622 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:53.972887 5716 raft_consensus.cc:2454] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:53.973338 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 35e34b60b78b402faf51e003c97db787 (DEFAULT_TABLE table=default.Table_aa4603034e7845b493166a810805b43c [id=71fd821e13734824a24f76fa2c220f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.973711 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e34b60b78b402faf51e003c97db787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.974627 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d (DEFAULT_TABLE table=default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.974727 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 79524df2415146ab926425c42068626c (DEFAULT_TABLE table=default.Table_b8acd47e87954187bc2d573c74b9290e [id=35093b3de3164b9c809aae8b5da3db54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.975157 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ec0526e1914221bc22b3e75ea01e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.975194 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.975870 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 (DEFAULT_TABLE table=default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.976374 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 753019050fcf4dc3b69722c3eb475037 (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.977088 5682 tablet_service.cc:1467] Processing CreateTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (DEFAULT_TABLE table=default.Table_c96ab339396440379b168db4805f8779 [id=746de1d0664945fd9a4a2cf4bed5405a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.977254 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4331375dd1664efba367672be446de48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:53.977389 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6c5e09f35af9471b89b6f760b282b361" } new_table_name: "default.Table_f69f572cb6f6424eb22ea998d9f6ba3f" modify_external_catalogs: true
W20240507 08:11:53.977828 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4331375dd1664efba367672be446de48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.978529 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 0a6a35991c18414b9bc714e9a0ba14d8 (table default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]) successfully deleted
I20240507 08:11:53.978758 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e0f3ee2b04de4cdf8a621b39f1476577 (table default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]) successfully deleted
W20240507 08:11:53.978978 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b4504b1ed3d449619926fe80469d9790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4504b1ed3d449619926fe80469d9790 already in progress: creating tablet
W20240507 08:11:53.979199 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5e53409fe13e4a9585dbba648a25cc07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e53409fe13e4a9585dbba648a25cc07 already in progress: creating tablet
W20240507 08:11:53.979429 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 88a2cfb5f046434c8340234a447a7e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a2cfb5f046434c8340234a447a7e65 already in progress: creating tablet
I20240507 08:11:53.980208 5689 tablet_service.cc:1467] Processing CreateTablet for tablet c43f3de489904f2ba2a637d408619bfc (DEFAULT_TABLE table=default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.980793 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_47712cc9868f40f9b5de852407b36598"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:53.981218 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47712cc9868f40f9b5de852407b36598 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:53.983893 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:53.988389 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037c3c0af5e464bb23a311e552fcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.988601 5563 tablet_service.cc:1467] Processing CreateTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 (DEFAULT_TABLE table=default.Table_c45ba94266f94dadbc071706db9038ff [id=6f1cedff6040494bb0a9bfe571648812]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.989463 5564 tablet_service.cc:1467] Processing CreateTablet for tablet c43f3de489904f2ba2a637d408619bfc (DEFAULT_TABLE table=default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:53.991744 5377 consensus_peers.cc:485] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9b8044e4a0a41218f67513b14c0ebeb. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:53.991961 5377 leader_election.cc:302] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:53.994633 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f22194aff40609a3cfa63b9b3aca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.995790 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 79524df2415146ab926425c42068626c (DEFAULT_TABLE table=default.Table_b8acd47e87954187bc2d573c74b9290e [id=35093b3de3164b9c809aae8b5da3db54]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:53.993394 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7c39ec9fec4331afabb39d1483ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.996181 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ec0526e1914221bc22b3e75ea01e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.996258 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79524df2415146ab926425c42068626c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.997025 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f3de489904f2ba2a637d408619bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.999377 5553 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 198) took 1088 ms. Trace:
I20240507 08:11:54.000924 5426 tablet_service.cc:1467] Processing CreateTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (DEFAULT_TABLE table=default.Table_c96ab339396440379b168db4805f8779 [id=746de1d0664945fd9a4a2cf4bed5405a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.001051 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:53.999501 5553 rpcz_store.cc:276] 0507 08:11:52.910418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.125311 (+214893us) service_pool.cc:225] Handling call
0507 08:11:53.162105 (+ 36794us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:11:53.162349 (+ 244us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:11:53.920784 (+758435us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.999180 (+ 78396us) tablet_replica.cc:205] Creating consensus instance
0507 08:11:53.999339 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:11:54.002404 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:53.994349 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 (DEFAULT_TABLE table=default.Table_dfc198b816c048acb4142d4ede67abb7 [id=3b2ccfca7fae432bb6acfa7e375f6ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.004715 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.005240 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79524df2415146ab926425c42068626c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.005336 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 975f22194aff40609a3cfa63b9b3aca2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.005717 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f294baf04bf74dfe8500cb8c1452d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.005990 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e34b60b78b402faf51e003c97db787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.006152 5994 raft_consensus.cc:2781] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:54.006359 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f3de489904f2ba2a637d408619bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.006556 5994 raft_consensus.cc:482] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:54.006675 5994 raft_consensus.cc:3037] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:54.000501 5839 consensus_queue.cc:260] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.009498 5994 raft_consensus.cc:504] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.010147 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 753019050fcf4dc3b69722c3eb475037. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.012295 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1c7c39ec9fec4331afabb39d1483ef57 because the tablet was not found. No further retry: Not found: Tablet not found: 1c7c39ec9fec4331afabb39d1483ef57
W20240507 08:11:54.012552 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 67c8624285d946e597227430a5714e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c8624285d946e597227430a5714e2f already in progress: creating tablet
I20240507 08:11:54.014984 5839 ts_tablet_manager.cc:1398] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.107s user 0.003s sys 0.000s
I20240507 08:11:54.015110 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_41b539df8107441aa4610f0fc94729c2" } modify_external_catalogs: true
I20240507 08:11:54.015323 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_41b539df8107441aa4610f0fc94729c2" } modify_external_catalogs: true
I20240507 08:11:54.015393 5553 tablet_service.cc:1467] Processing CreateTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (DEFAULT_TABLE table=default.Table_c96ab339396440379b168db4805f8779 [id=746de1d0664945fd9a4a2cf4bed5405a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.015765 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:54.020282 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:54.019520 5994 leader_election.cc:288] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:54.021745 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4c064b969844d938fddfe7f043c0b92 consensus state has no leader
W20240507 08:11:54.022224 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fcc1d7e309946d783a524b61d5f1269 consensus state has no leader
I20240507 08:11:54.022897 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.023125 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a425a90f0449bcaaacb4c3f653d26c consensus state has no leader
I20240507 08:11:54.024556 5841 log.cc:1196] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/34ffe98d424a4eb5a08be067f96a50f9
I20240507 08:11:54.025331 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4f4c310282004fbe930f8b2aa57ed2c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.025938 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f4c310282004fbe930f8b2aa57ed2c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.030433 5716 raft_consensus.cc:3037] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:54.033376 5716 raft_consensus.cc:2454] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:54.034415 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d (DEFAULT_TABLE table=default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.034637 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_26e7316a42a14b8fae7baa93114ebcd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:54.035068 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 79524df2415146ab926425c42068626c (DEFAULT_TABLE table=default.Table_b8acd47e87954187bc2d573c74b9290e [id=35093b3de3164b9c809aae8b5da3db54]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.035185 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e7316a42a14b8fae7baa93114ebcd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.036907 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_74a9e58e257645d6b826cab9e4134f48" } modify_external_catalogs: true
I20240507 08:11:54.036993 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 (DEFAULT_TABLE table=default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.037098 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_74a9e58e257645d6b826cab9e4134f48" } modify_external_catalogs: true
I20240507 08:11:54.040817 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8ad438587c5c48c78d151533b16a55e3" } modify_external_catalogs: true
I20240507 08:11:54.041024 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8ad438587c5c48c78d151533b16a55e3" } modify_external_catalogs: true
W20240507 08:11:54.044476 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0a0402c6efc8412aac0d21a545944073 consensus state has no leader
I20240507 08:11:54.045459 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.054307 5839 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.056388 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79524df2415146ab926425c42068626c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.058514 5378 consensus_peers.cc:485] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:54.058641 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 45997.5us
I20240507 08:11:54.059432 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_43a710eac1c44c56a13a40993d865715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.060148 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43a710eac1c44c56a13a40993d865715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.060883 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1648792bd6a04960aa1ed4847059e4b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:54.061115 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b8c75caecb914fe2afb9c345bba1be57" } new_table_name: "default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb" modify_external_catalogs: true
W20240507 08:11:54.061390 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1648792bd6a04960aa1ed4847059e4b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.062616 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef" } modify_external_catalogs: true
I20240507 08:11:54.062780 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef" } modify_external_catalogs: true
I20240507 08:11:54.063328 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:11:54.063896 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_487b0259f6ea4d48add257b1e7997937"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.064291 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_487b0259f6ea4d48add257b1e7997937 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.059967 5425 tablet_service.cc:1467] Processing CreateTablet for tablet c43f3de489904f2ba2a637d408619bfc (DEFAULT_TABLE table=default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.065447 6033 raft_consensus.cc:504] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.067831 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 35e34b60b78b402faf51e003c97db787 (DEFAULT_TABLE table=default.Table_aa4603034e7845b493166a810805b43c [id=71fd821e13734824a24f76fa2c220f1a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.075641 5554 tablet_service.cc:1467] Processing CreateTablet for tablet d451d03349794726a71c255a7f177436 (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.076088 5560 tablet_service.cc:1467] Processing CreateTablet for tablet b0b9f85cd78748d3afce83b64b3f823b (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.078302 5682 tablet_service.cc:1467] Processing CreateTablet for tablet b0b9f85cd78748d3afce83b64b3f823b (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.078438 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d03349794726a71c255a7f177436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.077636 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1037c3c0af5e464bb23a311e552fcf21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.079191 5699 tablet_service.cc:1467] Processing CreateTablet for tablet d451d03349794726a71c255a7f177436 (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.089632 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b9f85cd78748d3afce83b64b3f823b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.097594 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b9f85cd78748d3afce83b64b3f823b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.097939 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.100600 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44ec0526e1914221bc22b3e75ea01e0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.100634 5841 ts_tablet_manager.cc:1909] T 34ffe98d424a4eb5a08be067f96a50f9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.101512 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 111) took 2956 ms. Trace:
I20240507 08:11:54.102762 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43f3de489904f2ba2a637d408619bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.102701 5841 rpcz_store.cc:276] 0507 08:11:51.144653 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.148546 (+ 3893us) service_pool.cc:225] Handling call
0507 08:11:53.650048 (+2501502us) ts_tablet_manager.cc:1120] Deleting tablet 34ffe98d424a4eb5a08be067f96a50f9
0507 08:11:53.650059 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:53.707751 (+ 57692us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:53.713222 (+ 5471us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.101471 (+388249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2501318}
I20240507 08:11:54.103173 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 109) took 2982 ms. Trace:
I20240507 08:11:54.103291 5841 rpcz_store.cc:276] 0507 08:11:51.120320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.156035 (+ 35715us) service_pool.cc:225] Handling call
0507 08:11:54.103009 (+2946974us) ts_tablet_manager.cc:1120] Deleting tablet 97a43c27b30a4c3cbfd80dce5f28d7c7
0507 08:11:54.103019 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.103135 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945220}
I20240507 08:11:54.103587 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35e34b60b78b402faf51e003c97db787. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.103643 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 0d617213b959452ab6519809e56db128 (DEFAULT_TABLE table=default.Table_4c19e6eecd154d29a5b2cc7dae88a82a [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.103998 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d617213b959452ab6519809e56db128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.103585 5841 tablet_replica.cc:330] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.104549 5841 raft_consensus.cc:2229] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.104710 5841 raft_consensus.cc:2258] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.106619 5377 leader_election.cc:302] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:54.106869 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 72985us
W20240507 08:11:54.108492 5378 leader_election.cc:341] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:54.108997 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 50078.8us
I20240507 08:11:54.109084 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.109355 5424 tablet_service.cc:1467] Processing CreateTablet for tablet b0b9f85cd78748d3afce83b64b3f823b (DEFAULT_TABLE table=default.Table_13e0fb6057f044939108be222e8cb0f7 [id=8752e02495334c148e42bea97bf052b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.109620 5413 tablet_service.cc:1467] Processing CreateTablet for tablet d451d03349794726a71c255a7f177436 (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.109692 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0f3ee2b04de4cdf8a621b39f1476577" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:54.111716 5994 raft_consensus.cc:2781] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:54.111876 5994 raft_consensus.cc:686] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:54.113901 6033 leader_election.cc:288] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:54.117859 5550 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 214) took 1154 ms. Trace:
I20240507 08:11:54.124939 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d03349794726a71c255a7f177436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.127516 5994 consensus_queue.cc:237] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.132894 5841 ts_tablet_manager.cc:1875] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.133070 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0b9f85cd78748d3afce83b64b3f823b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.137265 5378 leader_election.cc:341] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:54.135951 5550 rpcz_store.cc:276] 0507 08:11:52.938089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.228086 (+289997us) service_pool.cc:225] Handling call
0507 08:11:53.293545 (+ 65459us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:11:53.294426 (+ 881us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:11:53.954470 (+660044us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.084133 (+129663us) tablet_replica.cc:205] Creating consensus instance
0507 08:11:54.092911 (+ 8778us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:11:54.162137 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_866be50aa9504f60904fd1dc4d383a17" } modify_external_catalogs: true
I20240507 08:11:54.162411 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_866be50aa9504f60904fd1dc4d383a17" } modify_external_catalogs: true
I20240507 08:11:54.183240 5841 ts_tablet_manager.cc:1888] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.191149 6049 raft_consensus.cc:482] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:54.191365 6049 raft_consensus.cc:504] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.192416 6049 leader_election.cc:288] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:54.196905 6049 raft_consensus.cc:482] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:54.197799 6049 raft_consensus.cc:504] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.198632 6049 leader_election.cc:288] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:54.199370 5520 leader_election.cc:341] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 120ebfd34a9e4d73a215cb9206013728
W20240507 08:11:54.199882 5520 leader_election.cc:341] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:54.201107 5841 log.cc:1196] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e0f3ee2b04de4cdf8a621b39f1476577
I20240507 08:11:54.208916 5841 ts_tablet_manager.cc:1909] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.209153 5843 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:54.209121 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 98aa4ec7b93943e5bf8f9758b3bb71e0
I20240507 08:11:54.209327 5843 ts_tablet_manager.cc:1367] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.397s user 0.019s sys 0.042s
I20240507 08:11:54.209837 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 113) took 2891 ms. Trace:
I20240507 08:11:54.210101 5843 raft_consensus.cc:348] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.211601 5843 raft_consensus.cc:374] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.210569 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.211697 5843 raft_consensus.cc:729] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.211656 5841 rpcz_store.cc:276] 0507 08:11:51.317900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.317978 (+ 78us) service_pool.cc:225] Handling call
0507 08:11:54.103509 (+2785531us) ts_tablet_manager.cc:1120] Deleting tablet e0f3ee2b04de4cdf8a621b39f1476577
0507 08:11:54.103518 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.165457 (+ 61939us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.183130 (+ 17673us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.209778 (+ 26648us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2782854}
I20240507 08:11:54.212023 5843 consensus_queue.cc:260] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.212551 5843 ts_tablet_manager.cc:1398] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:11:54.212831 5843 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:54.213759 5841 tablet_replica.cc:330] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.214419 5843 tablet_bootstrap.cc:654] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.214538 5841 raft_consensus.cc:2229] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.216070 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.215987 5841 raft_consensus.cc:2258] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.221604 5838 ts_tablet_manager.cc:1888] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.221755 5838 log.cc:1196] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/2cfa9a247eba402cb3f0b818ad9f837a
I20240507 08:11:54.222183 5838 ts_tablet_manager.cc:1909] T 2cfa9a247eba402cb3f0b818ad9f837a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.227141 5841 ts_tablet_manager.cc:1875] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.230978 5843 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:54.231137 5843 ts_tablet_manager.cc:1367] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240507 08:11:54.231918 5843 raft_consensus.cc:348] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.232126 5843 raft_consensus.cc:374] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.232214 5843 raft_consensus.cc:729] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.232426 5843 consensus_queue.cc:260] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.233369 5843 ts_tablet_manager.cc:1398] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:54.233722 5843 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:54.235574 5843 tablet_bootstrap.cc:654] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.236546 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 0d617213b959452ab6519809e56db128 (DEFAULT_TABLE table=default.Table_4c19e6eecd154d29a5b2cc7dae88a82a [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.236644 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 0d617213b959452ab6519809e56db128 (DEFAULT_TABLE table=default.Table_4c19e6eecd154d29a5b2cc7dae88a82a [id=bce0bd66e7b74db7934937410c81b20b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.242973 5517 leader_election.cc:341] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:54.243180 5517 leader_election.cc:302] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:11:54.243309 5377 leader_election.cc:341] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e0f3ee2b04de4cdf8a621b39f1476577
I20240507 08:11:54.243453 5377 leader_election.cc:302] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:54.243707 5377 raft_consensus.cc:2670] T e0f3ee2b04de4cdf8a621b39f1476577 P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:54.244664 5840 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
W20240507 08:11:54.244755 5517 leader_election.cc:341] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 120ebfd34a9e4d73a215cb9206013728
I20240507 08:11:54.244946 5517 leader_election.cc:302] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:54.245328 6049 raft_consensus.cc:2726] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:54.245765 6049 raft_consensus.cc:2726] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:54.246546 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.247452 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d451d03349794726a71c255a7f177436. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.247587 5840 ts_tablet_manager.cc:1367] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.518s user 0.007s sys 0.018s
I20240507 08:11:54.248243 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d617213b959452ab6519809e56db128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.250777 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 178) took 1678 ms. Trace:
I20240507 08:11:54.250916 5838 rpcz_store.cc:276] 0507 08:11:52.572236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.722943 (+150707us) service_pool.cc:225] Handling call
0507 08:11:53.842804 (+1119861us) ts_tablet_manager.cc:1120] Deleting tablet 2cfa9a247eba402cb3f0b818ad9f837a
0507 08:11:53.842812 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.049399 (+206587us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.221468 (+172069us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.250706 (+ 29238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109809}
I20240507 08:11:54.251474 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 183) took 1677 ms. Trace:
I20240507 08:11:54.251581 5838 rpcz_store.cc:276] 0507 08:11:52.573555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.739234 (+165679us) service_pool.cc:225] Handling call
0507 08:11:54.251263 (+1512029us) ts_tablet_manager.cc:1120] Deleting tablet 34e4e33f52df432b8d043171d7232c2a
0507 08:11:54.251274 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.251403 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495595}
I20240507 08:11:54.252044 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 185) took 1678 ms. Trace:
I20240507 08:11:54.252142 5838 rpcz_store.cc:276] 0507 08:11:52.573747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.755947 (+182200us) service_pool.cc:225] Handling call
0507 08:11:54.251837 (+1495890us) ts_tablet_manager.cc:1120] Deleting tablet c33710c6b55744c4ba0843c228abded3
0507 08:11:54.251847 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.251985 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495657}
I20240507 08:11:54.252487 5838 tablet_replica.cc:330] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.256035 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:54.257086 5377 consensus_peers.cc:485] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44f6f62c56cb4c29bf05914ca461a559. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:54.265413 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d617213b959452ab6519809e56db128. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.266726 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:54.266937 5840 raft_consensus.cc:348] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.268579 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3bf1c785cc9444d2a1e6dbbe3a4909f9" } new_table_name: "default.Table_59e07230b27a4e67acb956550768595d" modify_external_catalogs: true
W20240507 08:11:54.268889 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 907b5cb0e7a54903abe0b0d0e8d617dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 907b5cb0e7a54903abe0b0d0e8d617dc already in progress: creating tablet
W20240507 08:11:54.269150 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 already in progress: creating tablet
I20240507 08:11:54.269490 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b8acd47e87954187bc2d573c74b9290e" } new_table_name: "default.Table_16b866688308450cb4d4f81ccafa1afb" modify_external_catalogs: true
I20240507 08:11:54.269814 5839 tablet_bootstrap.cc:654] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.273036 5839 tablet_bootstrap.cc:492] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.273319 5839 ts_tablet_manager.cc:1367] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.258s user 0.005s sys 0.000s
I20240507 08:11:54.274060 5839 raft_consensus.cc:348] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.274268 5839 raft_consensus.cc:374] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.274917 5839 raft_consensus.cc:729] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.275436 5839 consensus_queue.cc:260] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.276892 5840 raft_consensus.cc:374] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.277408 5839 ts_tablet_manager.cc:1398] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:54.278725 5839 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.279250 5840 raft_consensus.cc:729] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.288023 5839 tablet_bootstrap.cc:654] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.288636 6064 tablet.cc:1722] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:54.285053 5840 consensus_queue.cc:260] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.289503 5974 raft_consensus.cc:2726] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:11:54.320210 5838 raft_consensus.cc:2229] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.320681 5838 raft_consensus.cc:2258] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.324040 5839 tablet_bootstrap.cc:492] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.324235 5839 ts_tablet_manager.cc:1367] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240507 08:11:54.328809 5994 raft_consensus.cc:482] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:54.329102 5994 raft_consensus.cc:504] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.330559 5994 leader_election.cc:288] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:54.331174 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10b38120dfe4b9ab3c39cf584e4ff13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:54.331446 5718 raft_consensus.cc:2454] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:54.331996 5377 leader_election.cc:302] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:54.332798 5994 raft_consensus.cc:2781] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:54.332988 5994 raft_consensus.cc:482] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:54.333168 5994 raft_consensus.cc:3037] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:54.335191 5839 raft_consensus.cc:348] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.335441 5839 raft_consensus.cc:374] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.335515 5839 raft_consensus.cc:729] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.335736 5839 consensus_queue.cc:260] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.336292 5839 ts_tablet_manager.cc:1398] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240507 08:11:54.336823 5994 raft_consensus.cc:504] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.338461 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10b38120dfe4b9ab3c39cf584e4ff13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:54.338728 5718 raft_consensus.cc:3037] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:54.339383 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 94647.3us
I20240507 08:11:54.342536 5718 raft_consensus.cc:2454] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:54.343037 5377 leader_election.cc:302] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:54.343356 6033 raft_consensus.cc:2781] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:54.343463 6033 raft_consensus.cc:686] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
W20240507 08:11:54.343612 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c33710c6b55744c4ba0843c228abded3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33710c6b55744c4ba0843c228abded3 already in progress: creating tablet
I20240507 08:11:54.343717 6033 consensus_queue.cc:237] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.344084 5839 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.350700 5839 tablet_bootstrap.cc:654] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.354629 5994 leader_election.cc:288] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:54.359019 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c4899c088531480ea15364af94595ee5
I20240507 08:11:54.359812 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4c19e6eecd154d29a5b2cc7dae88a82a" } new_table_name: "default.Table_5f853c15bb1145d8914412511c5172c1" modify_external_catalogs: true
I20240507 08:11:54.359987 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ac127de16ef340a4a2c4ac27753d230d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.360651 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac127de16ef340a4a2c4ac27753d230d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.361196 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f49dd4772f8e46478638b29c009c83ea
W20240507 08:11:54.361835 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4716c1285e91432a92b25ab13ada2779 consensus state has no leader
I20240507 08:11:54.361864 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16a4234bbc8e4c278d8ff310e4047b33" } new_table_name: "default.Table_2c361f4a301a46c98bcbb99d728d535f" modify_external_catalogs: true
I20240507 08:11:54.362150 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_342a9c864d774adf993c547b7e2f79d1" } modify_external_catalogs: true
I20240507 08:11:54.362330 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_342a9c864d774adf993c547b7e2f79d1" } modify_external_catalogs: true
W20240507 08:11:54.362756 5377 consensus_peers.cc:485] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 34e4e33f52df432b8d043171d7232c2a. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:54.363237 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fcf86fbc38ac4137b5b9acb214330443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.363852 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcf86fbc38ac4137b5b9acb214330443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.365803 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c43f3de489904f2ba2a637d408619bfc
I20240507 08:11:54.366537 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_329b8968956c49bdb8e75c6fb49ff3b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.366976 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_329b8968956c49bdb8e75c6fb49ff3b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.368341 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d73950d9f420413693022530757e7ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.368788 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d73950d9f420413693022530757e7ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.370517 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet c4899c088531480ea15364af94595ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.370965 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.371170 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.373850 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10b38120dfe4b9ab3c39cf584e4ff13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:54.374168 5838 ts_tablet_manager.cc:1875] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.374339 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d10b38120dfe4b9ab3c39cf584e4ff13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:54.376865 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03a59c7257e74b4abaa4ea7a1ee9136d consensus state has no leader
I20240507 08:11:54.377274 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d9766986d2d04989a90881e64f84e78e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.377789 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9766986d2d04989a90881e64f84e78e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.379137 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:54.379523 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79524df2415146ab926425c42068626c consensus state has no leader
I20240507 08:11:54.379655 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.379851 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet c4899c088531480ea15364af94595ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.380678 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dd26080257a24a50932dc596426fc19a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.381000 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd26080257a24a50932dc596426fc19a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.381666 5843 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:11:54.381994 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f49dd4772f8e46478638b29c009c83ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49dd4772f8e46478638b29c009c83ea already in progress: creating tablet
W20240507 08:11:54.382244 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c43f3de489904f2ba2a637d408619bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c43f3de489904f2ba2a637d408619bfc already in progress: creating tablet
W20240507 08:11:54.382462 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c4899c088531480ea15364af94595ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4899c088531480ea15364af94595ee5 already in progress: creating tablet
I20240507 08:11:54.382750 5843 ts_tablet_manager.cc:1367] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.149s user 0.013s sys 0.029s
I20240507 08:11:54.384014 5840 ts_tablet_manager.cc:1398] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.136s user 0.000s sys 0.008s
I20240507 08:11:54.384289 5843 raft_consensus.cc:348] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.384462 5840 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:54.384548 5843 raft_consensus.cc:374] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.384624 5843 raft_consensus.cc:729] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.384868 5843 consensus_queue.cc:260] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.385373 5843 ts_tablet_manager.cc:1398] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:11:54.385700 5843 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:54.387893 5843 tablet_bootstrap.cc:654] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.401697 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet c4899c088531480ea15364af94595ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.404613 5839 tablet_bootstrap.cc:492] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.404824 5839 ts_tablet_manager.cc:1367] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.000s
I20240507 08:11:54.405939 5839 raft_consensus.cc:348] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.406257 5839 raft_consensus.cc:374] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.406378 5839 raft_consensus.cc:729] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.406651 5839 consensus_queue.cc:260] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.407898 5839 ts_tablet_manager.cc:1398] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:54.408480 5839 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.410145 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:54.410287 5378 leader_election.cc:341] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:54.410656 5378 leader_election.cc:341] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:54.410912 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.414505 5838 ts_tablet_manager.cc:1888] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:54.414685 5838 log.cc:1196] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/baa842552d00489998f83e68aa0d516b
I20240507 08:11:54.415418 5838 ts_tablet_manager.cc:1909] T baa842552d00489998f83e68aa0d516b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.416502 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 199) took 1505 ms. Trace:
I20240507 08:11:54.416666 5838 rpcz_store.cc:276] 0507 08:11:52.910501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.151227 (+240726us) service_pool.cc:225] Handling call
0507 08:11:54.252394 (+1101167us) ts_tablet_manager.cc:1120] Deleting tablet baa842552d00489998f83e68aa0d516b
0507 08:11:54.252405 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.320498 (+ 68093us) spinlock_profiling.cc:243] Waited 67.5 ms on lock 0x6160000d1b54. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2af3ca00 00007f2f2c216b83 00007f2f2ecaa4ec 00007f2f2ecdc844 00007f2f2ecbdb30 00007f2f2ecbd8bd 0000000000a60bcc 00007f2f1f714b74 00007f2f1f71819b 00007f2f1f718018
0507 08:11:54.403690 (+ 83192us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.414236 (+ 10546us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.416386 (+ 2150us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":155185024,"tablet-delete.queue_time_us":1100364}
I20240507 08:11:54.418344 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet baa842552d00489998f83e68aa0d516b (table default.Table_5fc65aee40154e6296dc8b3f3076a4f3 [id=13e03112a7a641828153301cefc1774e]) successfully deleted
I20240507 08:11:54.420810 5839 tablet_bootstrap.cc:654] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:54.433264 5940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:11:54.433789 5940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:11:54.433990 5940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240507 08:11:54.434547 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 205) took 1518 ms. Trace:
I20240507 08:11:54.435168 5841 ts_tablet_manager.cc:1888] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.435354 5841 log.cc:1196] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e1cf2df0d3bd46268e22713fcc135a74
I20240507 08:11:54.435509 5838 rpcz_store.cc:276] 0507 08:11:52.911034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.164171 (+253137us) service_pool.cc:225] Handling call
0507 08:11:54.422195 (+1258024us) ts_tablet_manager.cc:1120] Deleting tablet 191adef34efa4c3999d4642a80177294
0507 08:11:54.422207 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.429102 (+ 6895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1222120}
I20240507 08:11:54.435763 5841 ts_tablet_manager.cc:1909] T e1cf2df0d3bd46268e22713fcc135a74 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:11:54.436183 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 191adef34efa4c3999d4642a80177294 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 191adef34efa4c3999d4642a80177294 already in progress: creating tablet
I20240507 08:11:54.436719 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 204) took 1525 ms. Trace:
I20240507 08:11:54.436743 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 114) took 3098 ms. Trace:
I20240507 08:11:54.437072 5838 rpcz_store.cc:276] 0507 08:11:52.910966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.162703 (+251737us) service_pool.cc:225] Handling call
0507 08:11:54.436539 (+1273836us) ts_tablet_manager.cc:1120] Deleting tablet 6bf34089e5ec419f97c0da9ec42d7eec
0507 08:11:54.436555 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.436680 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1230776}
I20240507 08:11:54.436945 5841 rpcz_store.cc:276] 0507 08:11:51.338679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.338730 (+ 51us) service_pool.cc:225] Handling call
0507 08:11:54.212630 (+2873900us) ts_tablet_manager.cc:1120] Deleting tablet e1cf2df0d3bd46268e22713fcc135a74
0507 08:11:54.213513 (+ 883us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.377236 (+163723us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.435027 (+ 57791us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.436704 (+ 1677us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2873683}
W20240507 08:11:54.437630 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6bf34089e5ec419f97c0da9ec42d7eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf34089e5ec419f97c0da9ec42d7eec already in progress: creating tablet
I20240507 08:11:54.438095 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 206) took 1526 ms. Trace:
I20240507 08:11:54.438167 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 125) took 3020 ms. Trace:
I20240507 08:11:54.438361 5841 rpcz_store.cc:276] 0507 08:11:51.417814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.417968 (+ 154us) service_pool.cc:225] Handling call
0507 08:11:54.438014 (+3020046us) ts_tablet_manager.cc:1120] Deleting tablet ea17889619fa420fbeacf1b41594c598
0507 08:11:54.438023 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.438127 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019796}
I20240507 08:11:54.438524 5838 rpcz_store.cc:276] 0507 08:11:52.911107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.194088 (+282981us) service_pool.cc:225] Handling call
0507 08:11:54.437939 (+1243851us) ts_tablet_manager.cc:1120] Deleting tablet add94bcf0c844cb78059b7240c7e28c0
0507 08:11:54.437952 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.438059 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196050}
W20240507 08:11:54.438808 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ea17889619fa420fbeacf1b41594c598 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea17889619fa420fbeacf1b41594c598 already in progress: creating tablet
W20240507 08:11:54.439021 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet add94bcf0c844cb78059b7240c7e28c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet add94bcf0c844cb78059b7240c7e28c0 already in progress: creating tablet
I20240507 08:11:54.439342 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 124) took 3022 ms. Trace:
I20240507 08:11:54.439424 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 216) took 1319 ms. Trace:
I20240507 08:11:54.440057 5841 rpcz_store.cc:276] 0507 08:11:51.416495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.447016 (+ 30521us) service_pool.cc:225] Handling call
0507 08:11:54.439193 (+2992177us) ts_tablet_manager.cc:1120] Deleting tablet 91c6cb7920df47aa97be6555c587798a
0507 08:11:54.439203 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.439308 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2991960}
W20240507 08:11:54.440536 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 91c6cb7920df47aa97be6555c587798a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91c6cb7920df47aa97be6555c587798a already in progress: creating tablet
I20240507 08:11:54.440874 5841 tablet_replica.cc:330] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.441000 5838 rpcz_store.cc:276] 0507 08:11:53.120267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.241919 (+121652us) service_pool.cc:225] Handling call
0507 08:11:54.439278 (+1197359us) ts_tablet_manager.cc:1120] Deleting tablet bde62374dc7f4a67a071b6b658e501c4
0507 08:11:54.439295 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.439390 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197193}
I20240507 08:11:54.441130 5841 raft_consensus.cc:2229] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:11:54.441416 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bde62374dc7f4a67a071b6b658e501c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bde62374dc7f4a67a071b6b658e501c4 already in progress: creating tablet
I20240507 08:11:54.441648 5841 raft_consensus.cc:2258] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.442193 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 217) took 1306 ms. Trace:
I20240507 08:11:54.442317 5838 rpcz_store.cc:276] 0507 08:11:53.135607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.242109 (+106502us) service_pool.cc:225] Handling call
0507 08:11:54.442067 (+1199958us) ts_tablet_manager.cc:1120] Deleting tablet cb7acbdb357445adb9ce69103c782a7f
0507 08:11:54.442075 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.442159 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199816}
W20240507 08:11:54.442741 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cb7acbdb357445adb9ce69103c782a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7acbdb357445adb9ce69103c782a7f already in progress: creating tablet
I20240507 08:11:54.442767 5841 ts_tablet_manager.cc:1875] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.443073 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 218) took 1304 ms. Trace:
I20240507 08:11:54.443188 5838 rpcz_store.cc:276] 0507 08:11:53.138459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.242261 (+103802us) service_pool.cc:225] Handling call
0507 08:11:54.442958 (+1200697us) ts_tablet_manager.cc:1120] Deleting tablet d10b38120dfe4b9ab3c39cf584e4ff13
0507 08:11:54.442966 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.443041 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1200572}
W20240507 08:11:54.443614 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d10b38120dfe4b9ab3c39cf584e4ff13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d10b38120dfe4b9ab3c39cf584e4ff13 already in progress: creating tablet
W20240507 08:11:54.444937 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d617213b959452ab6519809e56db128 consensus state has no leader
I20240507 08:11:54.445205 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 207) took 1534 ms. Trace:
I20240507 08:11:54.445350 5838 rpcz_store.cc:276] 0507 08:11:52.911184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.195995 (+284811us) service_pool.cc:225] Handling call
0507 08:11:54.445048 (+1249053us) ts_tablet_manager.cc:1120] Deleting tablet 80d82625304942d38b0261f95e235a41
0507 08:11:54.445061 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.445166 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1201186}
W20240507 08:11:54.445897 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 80d82625304942d38b0261f95e235a41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80d82625304942d38b0261f95e235a41 already in progress: creating tablet
I20240507 08:11:54.447621 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 220) took 1253 ms. Trace:
I20240507 08:11:54.447724 5838 rpcz_store.cc:276] 0507 08:11:53.194294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.243878 (+ 49584us) service_pool.cc:225] Handling call
0507 08:11:54.447486 (+1203608us) ts_tablet_manager.cc:1120] Deleting tablet 107d9b3f95e14044854ad0bf91a268ee
0507 08:11:54.447494 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.447591 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1203462}
W20240507 08:11:54.448194 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6f8f4205f1f48abaf4fcd7113ad44a8 consensus state has no leader
I20240507 08:11:54.449591 5841 ts_tablet_manager.cc:1888] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:54.449729 5841 log.cc:1196] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/98880a5ff9604cb4981c033c99e80be7
W20240507 08:11:54.449764 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 107d9b3f95e14044854ad0bf91a268ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107d9b3f95e14044854ad0bf91a268ee already in progress: creating tablet
I20240507 08:11:54.450021 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 907b5cb0e7a54903abe0b0d0e8d617dc from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.450134 5841 ts_tablet_manager.cc:1909] T 98880a5ff9604cb4981c033c99e80be7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.450325 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 221) took 1255 ms. Trace:
I20240507 08:11:54.450543 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 34bebec088a34ebbb06d2e707d54da4c from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.450449 5838 rpcz_store.cc:276] 0507 08:11:53.194495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.244032 (+ 49537us) service_pool.cc:225] Handling call
0507 08:11:54.450139 (+1206107us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:11:54.450151 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.450265 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205368}
I20240507 08:11:54.450918 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 128) took 2983 ms. Trace:
I20240507 08:11:54.451021 5841 rpcz_store.cc:276] 0507 08:11:51.467453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.467520 (+ 67us) service_pool.cc:225] Handling call
0507 08:11:54.440820 (+2973300us) ts_tablet_manager.cc:1120] Deleting tablet 98880a5ff9604cb4981c033c99e80be7
0507 08:11:54.440828 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.446111 (+ 5283us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.449460 (+ 3349us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.450862 (+ 1402us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973026}
I20240507 08:11:54.451572 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 127) took 2985 ms. Trace:
I20240507 08:11:54.451822 5841 rpcz_store.cc:276] 0507 08:11:51.466582 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.468492 (+ 1910us) service_pool.cc:225] Handling call
0507 08:11:54.451432 (+2982940us) ts_tablet_manager.cc:1120] Deleting tablet 98880a5ff9604cb4981c033c99e80be7
0507 08:11:54.451442 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.451520 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2982669}
I20240507 08:11:54.452394 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 126) took 3014 ms. Trace:
W20240507 08:11:54.452881 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 193ec0b463d04835b74df80229c85cde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 193ec0b463d04835b74df80229c85cde already in progress: creating tablet
I20240507 08:11:54.452906 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 98880a5ff9604cb4981c033c99e80be7 (table default.Table_11f043808d86473e9d1ff0c532d9c4c0 [id=245f6e995eca47f38bdc7495b40a6299]) successfully deleted
I20240507 08:11:54.453033 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 8ed8bac124cb4de983f0fdbaf728975f from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.452790 5841 rpcz_store.cc:276] 0507 08:11:51.438179 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.486891 (+ 48712us) service_pool.cc:225] Handling call
0507 08:11:54.452207 (+2965316us) ts_tablet_manager.cc:1120] Deleting tablet f29568ebe4774662a8b49b7829cf30e7
0507 08:11:54.452216 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.452344 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2965034}
W20240507 08:11:54.453625 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 98880a5ff9604cb4981c033c99e80be7 because the tablet was not found. No further retry: Not found: Tablet not found: 98880a5ff9604cb4981c033c99e80be7
I20240507 08:11:54.453835 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 208) took 1542 ms. Trace:
W20240507 08:11:54.453872 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f29568ebe4774662a8b49b7829cf30e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f29568ebe4774662a8b49b7829cf30e7 already in progress: creating tablet
I20240507 08:11:54.453892 5841 tablet_replica.cc:330] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.453970 5838 rpcz_store.cc:276] 0507 08:11:52.911271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.199725 (+288454us) service_pool.cc:225] Handling call
0507 08:11:54.453694 (+1253969us) ts_tablet_manager.cc:1120] Deleting tablet d537bed29a2f47058255d2530b5c0d23
0507 08:11:54.453707 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.453797 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199511}
I20240507 08:11:54.454232 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 7eaf0e2936324a9fbb2426a8026886fc from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.454277 5841 raft_consensus.cc:2229] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:54.454440 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d537bed29a2f47058255d2530b5c0d23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d537bed29a2f47058255d2530b5c0d23 already in progress: creating tablet
I20240507 08:11:54.454599 5841 raft_consensus.cc:2258] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.454737 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3bb889d1fe284f479791f644c016b27a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.455309 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bb889d1fe284f479791f644c016b27a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.455439 5841 ts_tablet_manager.cc:1875] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.457012 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet baa842552d00489998f83e68aa0d516b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.457077 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet d45d3fbbe3764261a3a24523a77c6443 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.457337 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 708d4bee30cc45bd9b86c56beb2c3ece with delete_type TABLET_DATA_DELETED (replaced by tablet 5228590c8e7d4c34b3acab9eb16665d3) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.457530 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 120ebfd34a9e4d73a215cb9206013728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:49 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.459187 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 98880a5ff9604cb4981c033c99e80be7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:54.459913 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 812dd1ebd42f4e929838a0535c22ad69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 812dd1ebd42f4e929838a0535c22ad69 already in progress: creating tablet
I20240507 08:11:54.451809 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:54.460944 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 168e49ae24314a7aae2291485ddeda44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168e49ae24314a7aae2291485ddeda44 already in progress: creating tablet
I20240507 08:11:54.461207 5837 tablet_replica.cc:330] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:54.387929 5840 tablet_bootstrap.cc:654] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.461416 5837 raft_consensus.cc:2229] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.461659 5837 raft_consensus.cc:2258] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:54.461877 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1b19d8e28f164c588946ea5e81ec320e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b19d8e28f164c588946ea5e81ec320e already in progress: creating tablet
I20240507 08:11:54.462424 5837 ts_tablet_manager.cc:1875] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.462594 5841 ts_tablet_manager.cc:1888] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.462780 5841 log.cc:1196] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/87e1f43426b14f98b985656f8d0d7141
W20240507 08:11:54.462994 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8e3ffaf1781e4b108f305346476905be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e3ffaf1781e4b108f305346476905be already in progress: creating tablet
I20240507 08:11:54.463266 5841 ts_tablet_manager.cc:1909] T 87e1f43426b14f98b985656f8d0d7141 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.463734 5838 tablet_replica.cc:330] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.463979 5838 raft_consensus.cc:2229] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.464162 5838 raft_consensus.cc:2258] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.464186 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 130) took 2950 ms. Trace:
I20240507 08:11:54.464380 5841 rpcz_store.cc:276] 0507 08:11:51.513933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.513989 (+ 56us) service_pool.cc:225] Handling call
0507 08:11:54.453814 (+2939825us) ts_tablet_manager.cc:1120] Deleting tablet 87e1f43426b14f98b985656f8d0d7141
0507 08:11:54.453822 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.459336 (+ 5514us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.462480 (+ 3144us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.464150 (+ 1670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2939559}
I20240507 08:11:54.464864 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 87e1f43426b14f98b985656f8d0d7141 (table default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]) successfully deleted
I20240507 08:11:54.464892 5838 ts_tablet_manager.cc:1875] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.465212 5841 tablet_replica.cc:330] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.465705 5841 raft_consensus.cc:2229] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:54.466075 5841 raft_consensus.cc:2258] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.466665 5841 ts_tablet_manager.cc:1875] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.468657 5837 ts_tablet_manager.cc:1888] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.468816 5837 log.cc:1196] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d10b38120dfe4b9ab3c39cf584e4ff13
I20240507 08:11:54.469237 5837 ts_tablet_manager.cc:1909] T d10b38120dfe4b9ab3c39cf584e4ff13 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:54.470266 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d10b38120dfe4b9ab3c39cf584e4ff13 (table default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]) successfully deleted
I20240507 08:11:54.472214 5840 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:54.472640 5840 ts_tablet_manager.cc:1367] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.088s user 0.001s sys 0.002s
I20240507 08:11:54.475428 5843 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:54.476161 5843 ts_tablet_manager.cc:1367] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.091s user 0.009s sys 0.004s
I20240507 08:11:54.476923 5843 raft_consensus.cc:348] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.477804 5840 raft_consensus.cc:348] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.478076 5843 raft_consensus.cc:374] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.478224 5843 raft_consensus.cc:729] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.476001 5841 ts_tablet_manager.cc:1888] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:54.478586 5841 log.cc:1196] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/87ede5b03251463d8608d15f037e4484
I20240507 08:11:54.479032 5841 ts_tablet_manager.cc:1909] T 87ede5b03251463d8608d15f037e4484 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.479838 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 129) took 2966 ms. Trace:
I20240507 08:11:54.480289 5841 rpcz_store.cc:276] 0507 08:11:51.513072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.514118 (+ 1046us) service_pool.cc:225] Handling call
0507 08:11:54.465157 (+2951039us) ts_tablet_manager.cc:1120] Deleting tablet 87ede5b03251463d8608d15f037e4484
0507 08:11:54.465166 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.470111 (+ 4945us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.475866 (+ 5755us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.479784 (+ 3918us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2950556}
I20240507 08:11:54.480595 5840 raft_consensus.cc:374] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.480721 5840 raft_consensus.cc:729] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.481106 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 87ede5b03251463d8608d15f037e4484 (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:11:54.480095 5843 consensus_queue.cc:260] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.484697 5843 ts_tablet_manager.cc:1398] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:11:54.485023 5840 consensus_queue.cc:260] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.485162 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ce35197b07dc454c82587e250579e404" } modify_external_catalogs: true
I20240507 08:11:54.485339 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ce35197b07dc454c82587e250579e404" } modify_external_catalogs: true
W20240507 08:11:54.485692 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 03a59c7257e74b4abaa4ea7a1ee9136d (table default.Table_59e07230b27a4e67acb956550768595d [id=ab3d4a967b1349a988bcd67feb187096], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03a59c7257e74b4abaa4ea7a1ee9136d consensus state has no leader
I20240507 08:11:54.485862 5841 tablet_replica.cc:330] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.486155 5841 raft_consensus.cc:2229] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.486351 5841 raft_consensus.cc:2258] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.486485 5259 catalog_manager.cc:5484] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:54.486873 5841 ts_tablet_manager.cc:1875] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.486990 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:54.488855 5422 tablet_service.cc:1467] Processing CreateTablet for tablet e298f11ff9f54e48b8de435f6849f39a (DEFAULT_TABLE table=default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.489261 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e298f11ff9f54e48b8de435f6849f39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.490960 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 33b62a463d8f416f8407dd866c5c993b (DEFAULT_TABLE table=default.Table_47712cc9868f40f9b5de852407b36598 [id=bb7a82da939941f6bdd819b36218f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.491312 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b62a463d8f416f8407dd866c5c993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.493854 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a (DEFAULT_TABLE table=default.Table_4fb1a5ebaf7c4b01bd7123a48782485f [id=0e33fa8f8fff4e09916ca2783a4903e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.494210 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e5a380f2784e3a806afcbb5cd2519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.497017 6073 raft_consensus.cc:482] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:54.497216 6073 raft_consensus.cc:504] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.497821 5424 tablet_service.cc:1467] Processing CreateTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.498191 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e68364238c4b66b8c0de0f4a2ea855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.500293 6073 leader_election.cc:288] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:54.501714 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet e1063aa5c7014d84b66f74ce28e3d890 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.501899 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 150efae7834740b798af86ae1dcfe445 (DEFAULT_TABLE table=default.Table_4331375dd1664efba367672be446de48 [id=d95a0f67e809437da87ad837fbf8f2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.502259 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150efae7834740b798af86ae1dcfe445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.507809 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 33b62a463d8f416f8407dd866c5c993b (DEFAULT_TABLE table=default.Table_47712cc9868f40f9b5de852407b36598 [id=bb7a82da939941f6bdd819b36218f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.508186 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b62a463d8f416f8407dd866c5c993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.508865 5415 tablet_service.cc:1467] Processing CreateTablet for tablet eb1901ddec07495abba085e7188c00d0 (DEFAULT_TABLE table=default.Table_2be2b9efedf84c4198bb1b2804f349d9 [id=b32e0b6aba9f46a497a2bcd3701a83eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.509228 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1901ddec07495abba085e7188c00d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.509268 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 34bebec088a34ebbb06d2e707d54da4c
I20240507 08:11:54.509786 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 34bebec088a34ebbb06d2e707d54da4c deleted and replaced by tablet 957775c01be94cc081db344c95a81310
W20240507 08:11:54.512851 5940 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 08:11:54.512964 5940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:11:54.513036 5940 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 08:11:54.513113 5940 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 08:11:54.513692 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.514034 5552 tablet_service.cc:1467] Processing CreateTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.507546 5520 leader_election.cc:341] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:54.514376 5555 tablet_service.cc:1467] Processing CreateTablet for tablet eb1901ddec07495abba085e7188c00d0 (DEFAULT_TABLE table=default.Table_2be2b9efedf84c4198bb1b2804f349d9 [id=b32e0b6aba9f46a497a2bcd3701a83eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.517987 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:11:54.518751 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:11:54.519133 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 7eaf0e2936324a9fbb2426a8026886fc deleted and replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:11:54.519433 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d45d3fbbe3764261a3a24523a77c6443
I20240507 08:11:54.519969 5263 catalog_manager.cc:6554] ReplaceTablet: tablet d45d3fbbe3764261a3a24523a77c6443 deleted and replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7
I20240507 08:11:54.520295 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 87377661d9374cef8ab1a97042b7f1b9
I20240507 08:11:54.522235 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.522611 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.526223 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.526893 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.534091 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 150efae7834740b798af86ae1dcfe445 (DEFAULT_TABLE table=default.Table_4331375dd1664efba367672be446de48 [id=d95a0f67e809437da87ad837fbf8f2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.534633 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150efae7834740b798af86ae1dcfe445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.538730 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e68364238c4b66b8c0de0f4a2ea855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.537927 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a (DEFAULT_TABLE table=default.Table_4fb1a5ebaf7c4b01bd7123a48782485f [id=0e33fa8f8fff4e09916ca2783a4903e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.537511 5560 tablet_service.cc:1467] Processing CreateTablet for tablet e298f11ff9f54e48b8de435f6849f39a (DEFAULT_TABLE table=default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.539331 5517 leader_election.cc:341] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:54.539579 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e298f11ff9f54e48b8de435f6849f39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.544183 5839 tablet_bootstrap.cc:492] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.544982 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e5a380f2784e3a806afcbb5cd2519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.546224 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:54.547044 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1901ddec07495abba085e7188c00d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.548624 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.489462 5692 tablet_service.cc:1467] Processing CreateTablet for tablet e298f11ff9f54e48b8de435f6849f39a (DEFAULT_TABLE table=default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.549212 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e298f11ff9f54e48b8de435f6849f39a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.489917 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 33b62a463d8f416f8407dd866c5c993b (DEFAULT_TABLE table=default.Table_47712cc9868f40f9b5de852407b36598 [id=bb7a82da939941f6bdd819b36218f7df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.550210 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 565ac6091f5e48b1b024094d5560e275 (DEFAULT_TABLE table=default.Table_3bb889d1fe284f479791f644c016b27a [id=796006aacf9741f8a4dc1d5916887655]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.550475 5517 leader_election.cc:302] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:54.550580 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33b62a463d8f416f8407dd866c5c993b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.490293 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a (DEFAULT_TABLE table=default.Table_4fb1a5ebaf7c4b01bd7123a48782485f [id=0e33fa8f8fff4e09916ca2783a4903e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.550902 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565ac6091f5e48b1b024094d5560e275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.551129 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e5a380f2784e3a806afcbb5cd2519a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.498548 5697 tablet_service.cc:1467] Processing CreateTablet for tablet eb1901ddec07495abba085e7188c00d0 (DEFAULT_TABLE table=default.Table_2be2b9efedf84c4198bb1b2804f349d9 [id=b32e0b6aba9f46a497a2bcd3701a83eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.552453 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1901ddec07495abba085e7188c00d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.552589 6073 raft_consensus.cc:2726] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:54.506104 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.506486 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 150efae7834740b798af86ae1dcfe445 (DEFAULT_TABLE table=default.Table_4331375dd1664efba367672be446de48 [id=d95a0f67e809437da87ad837fbf8f2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.510779 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.524296 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.525148 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.525352 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.525485 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.528245 6076 consensus_queue.cc:1035] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:54.561161 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 049483b0533642f2b46d8fa0e9b123cc (DEFAULT_TABLE table=default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.562505 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 34adee00b529424b9f274c61ce73b0bc (DEFAULT_TABLE table=default.Table_d9766986d2d04989a90881e64f84e78e [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.563338 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 2d45d93ed0d34806b9d9451c101f10af (DEFAULT_TABLE table=default.Table_26e7316a42a14b8fae7baa93114ebcd2 [id=6f00b78e5fce4f0181e4539ba6820c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.564386 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 (DEFAULT_TABLE table=default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.565243 5415 tablet_service.cc:1467] Processing CreateTablet for tablet b2df872d04b94c30872b03367df3cc31 (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.566143 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 31251576beee4a8a81a453b721c9c058 (DEFAULT_TABLE table=default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.566984 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049483b0533642f2b46d8fa0e9b123cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.567523 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34adee00b529424b9f274c61ce73b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.571503 5940 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.246.253
--webserver_port=40743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
I20240507 08:11:54.572213 5838 ts_tablet_manager.cc:1888] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.573922 5940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240507 08:11:54.575225 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d45d93ed0d34806b9d9451c101f10af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.576696 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_43a710eac1c44c56a13a40993d865715" } modify_external_catalogs: true
I20240507 08:11:54.576941 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_43a710eac1c44c56a13a40993d865715" } modify_external_catalogs: true
I20240507 08:11:54.578311 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_329b8968956c49bdb8e75c6fb49ff3b1" } modify_external_catalogs: true
I20240507 08:11:54.578488 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_329b8968956c49bdb8e75c6fb49ff3b1" } modify_external_catalogs: true
I20240507 08:11:54.579727 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_75283770e2c44851bcfa380f6385359d" } modify_external_catalogs: true
I20240507 08:11:54.579900 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_75283770e2c44851bcfa380f6385359d" } modify_external_catalogs: true
W20240507 08:11:54.582456 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 45389.6us
I20240507 08:11:54.582480 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3edbc94e25c549b2b444e45bac4bc8e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.583022 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3edbc94e25c549b2b444e45bac4bc8e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.584328 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4c69d38e484f5c819b7dd915bf4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.584383 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.584849 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.585294 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 31251576beee4a8a81a453b721c9c058 (DEFAULT_TABLE table=default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.585865 5546 tablet_service.cc:1467] Processing CreateTablet for tablet b2df872d04b94c30872b03367df3cc31 (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.586212 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2df872d04b94c30872b03367df3cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.587713 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3c344b571115426b99a5e43c1654596c" } new_table_name: "default.Table_44cd8bf73a544ecf89f263225686b5fd" modify_external_catalogs: true
I20240507 08:11:54.589043 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ac1bf080fe40499889f843cc9bf7e236"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.589491 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac1bf080fe40499889f843cc9bf7e236 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.589859 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 (DEFAULT_TABLE table=default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.590054 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 565ac6091f5e48b1b024094d5560e275 (DEFAULT_TABLE table=default.Table_3bb889d1fe284f479791f644c016b27a [id=796006aacf9741f8a4dc1d5916887655]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.590669 5378 consensus_peers.cc:485] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:54.590629 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f956db8e0c684a60a84296ba0504178d" } new_table_name: "default.Table_3b339252e2644a438c81a546b3cdb046" modify_external_catalogs: true
I20240507 08:11:54.590731 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565ac6091f5e48b1b024094d5560e275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.590919 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4c69d38e484f5c819b7dd915bf4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.592625 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_3c344b571115426b99a5e43c1654596c [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 1 got 0
I20240507 08:11:54.593040 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 34adee00b529424b9f274c61ce73b0bc (DEFAULT_TABLE table=default.Table_d9766986d2d04989a90881e64f84e78e [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.593165 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:11:54.593187 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31251576beee4a8a81a453b721c9c058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.593197 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2c00afce140541f39ea5236ad43ba1f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.593684 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c00afce140541f39ea5236ad43ba1f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.593767 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 049483b0533642f2b46d8fa0e9b123cc (DEFAULT_TABLE table=default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.594054 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 565ac6091f5e48b1b024094d5560e275 (DEFAULT_TABLE table=default.Table_3bb889d1fe284f479791f644c016b27a [id=796006aacf9741f8a4dc1d5916887655]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.594285 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.594452 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31251576beee4a8a81a453b721c9c058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.594477 5838 log.cc:1196] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5450ff9e72ec4462a38bcad2cd1aa5a4
I20240507 08:11:54.594864 5838 ts_tablet_manager.cc:1909] T 5450ff9e72ec4462a38bcad2cd1aa5a4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.595017 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2d45d93ed0d34806b9d9451c101f10af (DEFAULT_TABLE table=default.Table_26e7316a42a14b8fae7baa93114ebcd2 [id=6f00b78e5fce4f0181e4539ba6820c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.595315 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049483b0533642f2b46d8fa0e9b123cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.595731 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34adee00b529424b9f274c61ce73b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.596141 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 565ac6091f5e48b1b024094d5560e275. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.597262 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2df872d04b94c30872b03367df3cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.598131 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:11:54.598950 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.599233 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.600270 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b67c0c470e744311924705a8842f1926" } new_table_name: "default.Table_1c35bc706b22451f90611a3e7bc81f4d" modify_external_catalogs: true
I20240507 08:11:54.600765 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:11:54.601882 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.602330 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (DEFAULT_TABLE table=default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.602681 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.602895 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.603317 5557 tablet_service.cc:1467] Processing CreateTablet for tablet aa094773f3bf418492bcd5b5021fd429 (DEFAULT_TABLE table=default.Table_487b0259f6ea4d48add257b1e7997937 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.603657 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa094773f3bf418492bcd5b5021fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.603689 5940 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:11:54.605973 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (DEFAULT_TABLE table=default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.606375 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.609931 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.611017 5843 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:54.614560 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:54.617846 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.624845 5585 raft_consensus.cc:3037] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:54.625773 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 34bebec088a34ebbb06d2e707d54da4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34bebec088a34ebbb06d2e707d54da4c already in progress: creating tablet
W20240507 08:11:54.626042 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7eaf0e2936324a9fbb2426a8026886fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eaf0e2936324a9fbb2426a8026886fc already in progress: creating tablet
W20240507 08:11:54.626273 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d45d3fbbe3764261a3a24523a77c6443 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d45d3fbbe3764261a3a24523a77c6443 already in progress: creating tablet
I20240507 08:11:54.626248 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 150efae7834740b798af86ae1dcfe445. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.628970 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6e68364238c4b66b8c0de0f4a2ea855. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.629242 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 31251576beee4a8a81a453b721c9c058 (DEFAULT_TABLE table=default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.629751 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d45d93ed0d34806b9d9451c101f10af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.632272 5838 tablet_replica.cc:330] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.641263 5684 tablet_service.cc:1467] Processing CreateTablet for tablet b2df872d04b94c30872b03367df3cc31 (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.641762 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31251576beee4a8a81a453b721c9c058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.646106 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 (DEFAULT_TABLE table=default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.646307 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2df872d04b94c30872b03367df3cc31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.646292 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_44cd8bf73a544ecf89f263225686b5fd [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 2 got 0
I20240507 08:11:54.646780 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.648053 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b4c69d38e484f5c819b7dd915bf4410. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.649467 5426 tablet_service.cc:1467] Processing CreateTablet for tablet aa094773f3bf418492bcd5b5021fd429 (DEFAULT_TABLE table=default.Table_487b0259f6ea4d48add257b1e7997937 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.649952 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa094773f3bf418492bcd5b5021fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.653862 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b782575cf75a40b797e243b6a4059080" } new_table_name: "default.Table_b570163e3d2740c1abb9a44675630cfc" modify_external_catalogs: true
I20240507 08:11:54.655382 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16155f6d29ca4744902ede602ec6a8f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.655975 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16155f6d29ca4744902ede602ec6a8f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.656265 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c45ba94266f94dadbc071706db9038ff" } new_table_name: "default.Table_6d8668c1f9b94e15b7b7603a49f11a7d" modify_external_catalogs: true
I20240507 08:11:54.657727 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_38102920150a4a08a922cdb41801b972"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.658190 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1c6b488b9014841899597d858bda5e2 consensus state has no leader
W20240507 08:11:54.658224 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38102920150a4a08a922cdb41801b972 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.658926 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4d38bf47908640beb3f6d94ad4446717"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.659368 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d38bf47908640beb3f6d94ad4446717 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.660197 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 049483b0533642f2b46d8fa0e9b123cc (DEFAULT_TABLE table=default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.662429 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 34adee00b529424b9f274c61ce73b0bc (DEFAULT_TABLE table=default.Table_d9766986d2d04989a90881e64f84e78e [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.664084 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 2d45d93ed0d34806b9d9451c101f10af (DEFAULT_TABLE table=default.Table_26e7316a42a14b8fae7baa93114ebcd2 [id=6f00b78e5fce4f0181e4539ba6820c24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.664749 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.665136 5839 ts_tablet_manager.cc:1367] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.139s user 0.018s sys 0.027s
W20240507 08:11:54.665405 5654 consensus_peers.cc:485] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:54.666994 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7ba2e8d1e01047fd9cc1c192e9628750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba2e8d1e01047fd9cc1c192e9628750 already in progress: creating tablet
W20240507 08:11:54.667284 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 87377661d9374cef8ab1a97042b7f1b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87377661d9374cef8ab1a97042b7f1b9 already in progress: creating tablet
I20240507 08:11:54.667822 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (DEFAULT_TABLE table=default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.669374 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.669418 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.669636 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.669795 5696 tablet_service.cc:1467] Processing CreateTablet for tablet aa094773f3bf418492bcd5b5021fd429 (DEFAULT_TABLE table=default.Table_487b0259f6ea4d48add257b1e7997937 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.670076 5683 tablet_service.cc:1467] Processing CreateTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc (DEFAULT_TABLE table=default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:54.670799 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a56cb45381604ac9a69e6f3edaa4eefc because the tablet was not found. No further retry: Not found: Tablet not found: a56cb45381604ac9a69e6f3edaa4eefc
W20240507 08:11:54.672699 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1cd02a49359743c4b4e540ab0338d8ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cd02a49359743c4b4e540ab0338d8ed already in progress: creating tablet
W20240507 08:11:54.678871 6092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:11:54.680140 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f because the tablet was not found. No further retry: Not found: Tablet not found: 3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:11:54.683081 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 049483b0533642f2b46d8fa0e9b123cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.688534 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c2e5247aef1b421c8e538b0796249648"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.689141 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2e5247aef1b421c8e538b0796249648 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.689782 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56cb45381604ac9a69e6f3edaa4eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.691329 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d45d93ed0d34806b9d9451c101f10af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.691577 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34adee00b529424b9f274c61ce73b0bc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.691571 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 58050.5us
I20240507 08:11:54.693871 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa094773f3bf418492bcd5b5021fd429. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.694109 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.694185 5427 tablet_service.cc:1467] Processing CreateTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc (DEFAULT_TABLE table=default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.694602 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56cb45381604ac9a69e6f3edaa4eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.698444 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet f29568ebe4774662a8b49b7829cf30e7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.697888 5839 raft_consensus.cc:348] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.699020 5839 raft_consensus.cc:374] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.699488 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 9211028e9eeb452d8ab5903710349ca6 (DEFAULT_TABLE table=default.Table_1648792bd6a04960aa1ed4847059e4b6 [id=1bd93ca3e4604ef7b1f2a16e796c8862]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.699476 5839 raft_consensus.cc:729] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.700917 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e850547c67ff4a8fa952777edc77bd59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:54.701480 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d8b43861f60b4b6189e6386d097240a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:54.701684 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211028e9eeb452d8ab5903710349ca6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:54.701952 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8b43861f60b4b6189e6386d097240a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.701321 5839 consensus_queue.cc:260] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:54.702407 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e850547c67ff4a8fa952777edc77bd59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.705753 5839 ts_tablet_manager.cc:1398] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20240507 08:11:54.708499 5839 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.710470 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5450ff9e72ec4462a38bcad2cd1aa5a4 (table default.Table_5a63960776644f8186340cc40315f99f [id=fbffc17394064c7db3473a8898f2360f]) successfully deleted
W20240507 08:11:54.711412 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b22334378eec4ab3885d0f9d2d812a3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22334378eec4ab3885d0f9d2d812a3e already in progress: creating tablet
W20240507 08:11:54.712323 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8ed8bac124cb4de983f0fdbaf728975f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed8bac124cb4de983f0fdbaf728975f already in progress: creating tablet
W20240507 08:11:54.713065 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 697611671ef64c53b0be4f5341b8f270 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 697611671ef64c53b0be4f5341b8f270 already in progress: creating tablet
W20240507 08:11:54.715653 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4867e1cca1634b4ab6e36f9ed3204e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4867e1cca1634b4ab6e36f9ed3204e0a already in progress: creating tablet
W20240507 08:11:54.717561 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b4504b1ed3d449619926fe80469d9790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4504b1ed3d449619926fe80469d9790 already in progress: creating tablet
W20240507 08:11:54.718865 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 88a2cfb5f046434c8340234a447a7e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a2cfb5f046434c8340234a447a7e65 already in progress: creating tablet
W20240507 08:11:54.720381 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 67c8624285d946e597227430a5714e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c8624285d946e597227430a5714e2f already in progress: creating tablet
W20240507 08:11:54.720664 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5e53409fe13e4a9585dbba648a25cc07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e53409fe13e4a9585dbba648a25cc07 already in progress: creating tablet
W20240507 08:11:54.721045 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1c7c39ec9fec4331afabb39d1483ef57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c7c39ec9fec4331afabb39d1483ef57 already in progress: creating tablet
W20240507 08:11:54.721276 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 907b5cb0e7a54903abe0b0d0e8d617dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 907b5cb0e7a54903abe0b0d0e8d617dc already in progress: creating tablet
W20240507 08:11:54.721554 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 already in progress: creating tablet
W20240507 08:11:54.721803 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c4899c088531480ea15364af94595ee5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4899c088531480ea15364af94595ee5 already in progress: creating tablet
W20240507 08:11:54.722061 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c43f3de489904f2ba2a637d408619bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c43f3de489904f2ba2a637d408619bfc already in progress: creating tablet
W20240507 08:11:54.722498 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f49dd4772f8e46478638b29c009c83ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49dd4772f8e46478638b29c009c83ea already in progress: creating tablet
W20240507 08:11:54.723112 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet baa842552d00489998f83e68aa0d516b because the tablet was not found. No further retry: Not found: Tablet not found: baa842552d00489998f83e68aa0d516b
I20240507 08:11:54.723690 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 9211028e9eeb452d8ab5903710349ca6 (DEFAULT_TABLE table=default.Table_1648792bd6a04960aa1ed4847059e4b6 [id=1bd93ca3e4604ef7b1f2a16e796c8862]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.724180 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211028e9eeb452d8ab5903710349ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.725448 5549 tablet_service.cc:1467] Processing CreateTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc (DEFAULT_TABLE table=default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.725878 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a56cb45381604ac9a69e6f3edaa4eefc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.730489 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 9211028e9eeb452d8ab5903710349ca6 (DEFAULT_TABLE table=default.Table_1648792bd6a04960aa1ed4847059e4b6 [id=1bd93ca3e4604ef7b1f2a16e796c8862]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:54.730870 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9211028e9eeb452d8ab5903710349ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:54.739729 5839 tablet_bootstrap.cc:654] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:54.779708 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 54103.5us
I20240507 08:11:54.791584 5839 tablet_bootstrap.cc:492] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.793917 6099 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:54.794636 5839 ts_tablet_manager.cc:1367] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.004s
I20240507 08:11:54.794889 5838 raft_consensus.cc:2229] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.795224 5838 raft_consensus.cc:2258] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.796216 5838 ts_tablet_manager.cc:1875] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.795444 5839 raft_consensus.cc:348] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.796701 5839 raft_consensus.cc:374] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.796861 5839 raft_consensus.cc:729] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.797310 5839 consensus_queue.cc:260] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.800881 5839 ts_tablet_manager.cc:1398] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:54.803081 5840 ts_tablet_manager.cc:1398] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.326s user 0.002s sys 0.000s
I20240507 08:11:54.804208 5839 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.811820 5838 ts_tablet_manager.cc:1888] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:54.811970 5838 log.cc:1196] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/708d4bee30cc45bd9b86c56beb2c3ece
I20240507 08:11:54.812445 5838 ts_tablet_manager.cc:1909] T 708d4bee30cc45bd9b86c56beb2c3ece P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.813069 5838 tablet_replica.cc:330] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.813294 5838 raft_consensus.cc:2229] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.813481 5838 raft_consensus.cc:2258] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.814079 5838 ts_tablet_manager.cc:1875] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.827064 5843 tablet_bootstrap.cc:654] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.837710 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_97792a6d740f4599b202087719f29978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:54.838063 5840 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:54.838543 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97792a6d740f4599b202087719f29978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.840174 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ac127de16ef340a4a2c4ac27753d230d" } modify_external_catalogs: true
I20240507 08:11:54.840366 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ac127de16ef340a4a2c4ac27753d230d" } modify_external_catalogs: true
W20240507 08:11:54.840945 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4306751d16444ec5948c2ba5831063f4 consensus state has no leader
I20240507 08:11:54.841061 5840 tablet_bootstrap.cc:654] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.841549 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet b6e68364238c4b66b8c0de0f4a2ea855 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.841387 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e85104a6436841ad8d9510f2e54233c3" } modify_external_catalogs: true
I20240507 08:11:54.841837 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e85104a6436841ad8d9510f2e54233c3" } modify_external_catalogs: true
I20240507 08:11:54.844000 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_efefbeaaab494b34a0acd79bec1f4185" } modify_external_catalogs: true
I20240507 08:11:54.844328 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_efefbeaaab494b34a0acd79bec1f4185" } modify_external_catalogs: true
W20240507 08:11:54.844847 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 070fdbf0b00b4582a184729c2f2a4b69 consensus state has no leader
I20240507 08:11:54.845364 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71fb81b62c8746bf9376dd00c4250d3a" } modify_external_catalogs: true
I20240507 08:11:54.845499 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71fb81b62c8746bf9376dd00c4250d3a" } modify_external_catalogs: true
W20240507 08:11:54.849417 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f294baf04bf74dfe8500cb8c1452d8b3 consensus state has no leader
I20240507 08:11:54.850231 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bc44d932cc29477e8751f038712ef26c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.850947 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc44d932cc29477e8751f038712ef26c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.853196 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 708d4bee30cc45bd9b86c56beb2c3ece (table default.Table_eff2f49d93764317a595c308278c0732 [id=2b718529a9e7440d831c3d60a8c17ba4]) successfully deleted
I20240507 08:11:54.854027 5839 tablet_bootstrap.cc:654] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.858537 5838 ts_tablet_manager.cc:1888] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.858848 5838 log.cc:1196] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/120ebfd34a9e4d73a215cb9206013728
I20240507 08:11:54.858969 5843 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:54.859253 5838 ts_tablet_manager.cc:1909] T 120ebfd34a9e4d73a215cb9206013728 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.859313 5843 ts_tablet_manager.cc:1367] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.248s user 0.000s sys 0.005s
I20240507 08:11:54.859807 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c076159e489f4d5a97e67533975f0a43" } modify_external_catalogs: true
I20240507 08:11:54.859970 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c076159e489f4d5a97e67533975f0a43" } modify_external_catalogs: true
I20240507 08:11:54.860359 5843 raft_consensus.cc:348] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.860654 5843 raft_consensus.cc:374] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.860738 5843 raft_consensus.cc:729] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.860808 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6d066df70e044e5d8fab4f4806216b98" } new_table_name: "default.Table_8cc125a833524bb3b0fef55c817ffe0e" modify_external_catalogs: true
I20240507 08:11:54.861057 5843 consensus_queue.cc:260] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.861697 5838 tablet_replica.cc:330] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.861881 5838 raft_consensus.cc:2229] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.862058 5838 raft_consensus.cc:2258] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.863914 5838 ts_tablet_manager.cc:1875] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.865479 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4331375dd1664efba367672be446de48" } new_table_name: "default.Table_1d73a8ee5a4848628ed2472bdc0b084b" modify_external_catalogs: true
W20240507 08:11:54.866039 6096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:54.867691 5839 tablet_bootstrap.cc:492] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:54.867916 5839 ts_tablet_manager.cc:1367] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.001s
I20240507 08:11:54.869029 5839 raft_consensus.cc:348] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.869338 5839 raft_consensus.cc:374] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.869431 5839 raft_consensus.cc:729] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.869834 5839 consensus_queue.cc:260] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.870268 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0074cc70e248ac8b5e90cf57b441c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.871920 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 87ede5b03251463d8608d15f037e4484 with delete_type TABLET_DATA_DELETED (replaced by tablet 91c6cb7920df47aa97be6555c587798a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.872308 5839 ts_tablet_manager.cc:1398] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240507 08:11:54.872713 5839 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:54.875741 5839 tablet_bootstrap.cc:654] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:54.877774 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 120ebfd34a9e4d73a215cb9206013728 (table default.Table_03d2a58624f744daaa63f01a2b08b60e [id=4fb86dd0d8a542a29dad10c878bdee2f]) successfully deleted
W20240507 08:11:54.882381 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b2de528d9fc046f183f70388cffab6c8 consensus state has no leader
I20240507 08:11:54.884519 5838 ts_tablet_manager.cc:1888] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:11:54.885190 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 150efae7834740b798af86ae1dcfe445 consensus state has no leader
I20240507 08:11:54.885174 5838 log.cc:1196] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/98880a5ff9604cb4981c033c99e80be7
I20240507 08:11:54.885560 5838 ts_tablet_manager.cc:1909] T 98880a5ff9604cb4981c033c99e80be7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.886744 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_be493cfc9faf43faa5b0a4afe7287d6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:54.887296 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be493cfc9faf43faa5b0a4afe7287d6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:54.891286 5838 tablet_replica.cc:330] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.891487 5838 raft_consensus.cc:2229] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.891638 5838 raft_consensus.cc:2258] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.892184 5838 ts_tablet_manager.cc:1875] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.893193 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b6e68364238c4b66b8c0de0f4a2ea855
I20240507 08:11:54.894344 5259 catalog_manager.cc:6554] ReplaceTablet: tablet b6e68364238c4b66b8c0de0f4a2ea855 deleted and replaced by tablet 8a015bf84a454d49b6497c16ed15c84a
I20240507 08:11:54.896448 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 5228590c8e7d4c34b3acab9eb16665d3 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.897505 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c48da19c9b9e4207b267137d240adc5b
I20240507 08:11:54.898339 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:54.898591 5838 ts_tablet_manager.cc:1888] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.898720 5838 log.cc:1196] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e1063aa5c7014d84b66f74ce28e3d890
I20240507 08:11:54.899091 5838 ts_tablet_manager.cc:1909] T e1063aa5c7014d84b66f74ce28e3d890 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:11:54.899202 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c48da19c9b9e4207b267137d240adc5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48da19c9b9e4207b267137d240adc5b already in progress: creating tablet
I20240507 08:11:54.900063 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:54.900763 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b6e68364238c4b66b8c0de0f4a2ea855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6e68364238c4b66b8c0de0f4a2ea855 already in progress: creating tablet
I20240507 08:11:54.902442 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ea5220905954dc9b078e00e0db69fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.902781 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.903097 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.904235 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.906839 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 98880a5ff9604cb4981c033c99e80be7 (table default.Table_11f043808d86473e9d1ff0c532d9c4c0 [id=245f6e995eca47f38bdc7495b40a6299]) successfully deleted
I20240507 08:11:54.907039 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e1063aa5c7014d84b66f74ce28e3d890 (table default.Table_1eb6ce65380844798cfe02364195022d [id=3e07ae7d381f4b168b86a0b4d8d93765]) successfully deleted
W20240507 08:11:54.907279 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 87377661d9374cef8ab1a97042b7f1b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87377661d9374cef8ab1a97042b7f1b9 already in progress: creating tablet
W20240507 08:11:54.907521 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d45d3fbbe3764261a3a24523a77c6443 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d45d3fbbe3764261a3a24523a77c6443 already in progress: creating tablet
W20240507 08:11:54.907735 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a56cb45381604ac9a69e6f3edaa4eefc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a56cb45381604ac9a69e6f3edaa4eefc already in progress: creating tablet
W20240507 08:11:54.907955 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f already in progress: creating tablet
W20240507 08:11:54.908164 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1cd02a49359743c4b4e540ab0338d8ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cd02a49359743c4b4e540ab0338d8ed already in progress: creating tablet
I20240507 08:11:54.908421 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet c33710c6b55744c4ba0843c228abded3 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.911310 5840 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:54.911574 5840 ts_tablet_manager.cc:1367] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.004s
I20240507 08:11:54.912703 5840 raft_consensus.cc:348] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.913683 5841 ts_tablet_manager.cc:1888] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:54.913848 5841 log.cc:1196] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/ed0074cc70e248ac8b5e90cf57b441c3
I20240507 08:11:54.914824 5841 ts_tablet_manager.cc:1909] T ed0074cc70e248ac8b5e90cf57b441c3 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.915117 5255 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_44cd8bf73a544ecf89f263225686b5fd [id=e72c5d8cfeb14c3b9ae7d8c6e1499677] alter complete (version 2)
I20240507 08:11:54.914741 5840 raft_consensus.cc:374] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.916127 5840 raft_consensus.cc:729] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.914791 5838 tablet_replica.cc:330] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.916442 5840 consensus_queue.cc:260] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:54.916673 5838 raft_consensus.cc:2229] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:54.916754 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7ba2e8d1e01047fd9cc1c192e9628750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba2e8d1e01047fd9cc1c192e9628750 already in progress: creating tablet
W20240507 08:11:54.916990 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7eaf0e2936324a9fbb2426a8026886fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7eaf0e2936324a9fbb2426a8026886fc already in progress: creating tablet
W20240507 08:11:54.917188 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 34bebec088a34ebbb06d2e707d54da4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34bebec088a34ebbb06d2e707d54da4c already in progress: creating tablet
I20240507 08:11:54.919495 5838 raft_consensus.cc:2258] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.920359 5838 ts_tablet_manager.cc:1875] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.923352 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_44cd8bf73a544ecf89f263225686b5fd [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 2 got 0
I20240507 08:11:54.927158 6100 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:54.930159 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 132) took 3411 ms. Trace:
I20240507 08:11:54.930517 5841 rpcz_store.cc:276] 0507 08:11:51.518479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.518534 (+ 55us) service_pool.cc:225] Handling call
0507 08:11:54.485371 (+2966837us) ts_tablet_manager.cc:1120] Deleting tablet ed0074cc70e248ac8b5e90cf57b441c3
0507 08:11:54.485381 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.844251 (+358870us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.907243 (+ 62992us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.930109 (+ 22866us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14433,"tablet-delete.queue_time_us":2966628}
I20240507 08:11:54.930908 5841 tablet_replica.cc:330] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.931102 5841 raft_consensus.cc:2229] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:54.931443 5841 raft_consensus.cc:2258] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.931669 5838 ts_tablet_manager.cc:1888] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.931782 5838 log.cc:1196] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/ed0074cc70e248ac8b5e90cf57b441c3
I20240507 08:11:54.932124 5838 ts_tablet_manager.cc:1909] T ed0074cc70e248ac8b5e90cf57b441c3 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.932821 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet ed0074cc70e248ac8b5e90cf57b441c3 (table default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]) successfully deleted
I20240507 08:11:54.933192 5841 ts_tablet_manager.cc:1875] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.934980 5843 ts_tablet_manager.cc:1398] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.075s user 0.000s sys 0.002s
I20240507 08:11:54.935335 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet ed0074cc70e248ac8b5e90cf57b441c3 (table default.Table_50c7cb69774c461e88d7610b086cc0ba [id=fd302d9a47234b1e821d1bbfe9bee40a]) successfully deleted
I20240507 08:11:54.935716 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.936376 5838 tablet_replica.cc:330] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.937911 5838 raft_consensus.cc:2229] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.938252 5838 raft_consensus.cc:2258] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.940986 5840 ts_tablet_manager.cc:1398] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240507 08:11:54.941092 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:54.941707 5838 ts_tablet_manager.cc:1875] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.943181 5840 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:54.945508 5841 ts_tablet_manager.cc:1888] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:54.946208 5995 consensus_queue.cc:1035] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:54.947969 5841 log.cc:1196] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/708d4bee30cc45bd9b86c56beb2c3ece
I20240507 08:11:54.948434 5841 ts_tablet_manager.cc:1909] T 708d4bee30cc45bd9b86c56beb2c3ece P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:54.946477 6033 raft_consensus.cc:482] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:54.951726 6033 raft_consensus.cc:504] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.952680 6033 leader_election.cc:288] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:54.954305 5838 ts_tablet_manager.cc:1888] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.954442 5838 log.cc:1196] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/87ede5b03251463d8608d15f037e4484
I20240507 08:11:54.954774 5838 ts_tablet_manager.cc:1909] T 87ede5b03251463d8608d15f037e4484 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:11:54.955399 5377 leader_election.cc:341] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 91c6cb7920df47aa97be6555c587798a
I20240507 08:11:54.955711 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 87ede5b03251463d8608d15f037e4484 (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:11:54.956437 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c6cb7920df47aa97be6555c587798a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:54.957509 5378 leader_election.cc:341] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:54.957669 5378 leader_election.cc:302] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:54.958194 6033 raft_consensus.cc:2726] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:54.958503 5838 tablet_replica.cc:330] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:54.959017 5838 raft_consensus.cc:2229] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:54.959415 5838 raft_consensus.cc:2258] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.960243 5838 ts_tablet_manager.cc:1875] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:54.961475 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 131) took 3443 ms. Trace:
I20240507 08:11:54.962633 5841 rpcz_store.cc:276] 0507 08:11:51.517997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.518871 (+ 874us) service_pool.cc:225] Handling call
0507 08:11:54.930838 (+3411967us) ts_tablet_manager.cc:1120] Deleting tablet 708d4bee30cc45bd9b86c56beb2c3ece
0507 08:11:54.930847 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:54.938950 (+ 8103us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.945406 (+ 6456us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:54.961392 (+ 15986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411768}
I20240507 08:11:54.963496 5841 tablet_replica.cc:330] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:54.965107 5841 raft_consensus.cc:2229] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:54.966029 5841 pending_rounds.cc:62] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:54.966645 5841 pending_rounds.cc:68] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924725109915648 op_type: NO_OP noop_request { }
I20240507 08:11:54.967037 5838 ts_tablet_manager.cc:1888] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:54.967159 5838 log.cc:1196] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/8ea5220905954dc9b078e00e0db69fd2
I20240507 08:11:54.967489 5838 ts_tablet_manager.cc:1909] T 8ea5220905954dc9b078e00e0db69fd2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:54.968447 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 8ea5220905954dc9b078e00e0db69fd2 (table default.Table_b650283bfd9b448c8a0ba81f514f2831 [id=a2716884f32e46468e513117cd78b7af]) successfully deleted
W20240507 08:11:54.969101 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b6e68364238c4b66b8c0de0f4a2ea855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6e68364238c4b66b8c0de0f4a2ea855 already in progress: creating tablet
I20240507 08:11:54.969525 5839 tablet_bootstrap.cc:492] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:11:54.969687 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c48da19c9b9e4207b267137d240adc5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48da19c9b9e4207b267137d240adc5b already in progress: creating tablet
I20240507 08:11:54.969817 5839 ts_tablet_manager.cc:1367] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.002s
I20240507 08:11:54.971030 5839 raft_consensus.cc:348] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:54.971323 5839 raft_consensus.cc:374] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:54.971439 5839 raft_consensus.cc:729] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:54.967504 5841 raft_consensus.cc:2866] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:54.972195 5841 raft_consensus.cc:2258] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:54.973002 5841 ts_tablet_manager.cc:1875] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:54.979558 5378 consensus_peers.cc:485] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:54.980466 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:54.982765 5843 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:54.985517 5843 tablet_bootstrap.cc:654] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:54.989357 6110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:11:54.993803 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet dfa14901cdf14b72bea365b04ad521bd from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:54.998889 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet dfa14901cdf14b72bea365b04ad521bd
I20240507 08:11:54.999382 5255 catalog_manager.cc:6554] ReplaceTablet: tablet dfa14901cdf14b72bea365b04ad521bd deleted and replaced by tablet 3f01e3311a8b4553a31914922fb08d13
I20240507 08:11:54.999706 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:54.999804 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.000955 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:55.001535 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet dfa14901cdf14b72bea365b04ad521bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfa14901cdf14b72bea365b04ad521bd already in progress: creating tablet
I20240507 08:11:54.971741 5839 consensus_queue.cc:260] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:55.001740 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet dfa14901cdf14b72bea365b04ad521bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfa14901cdf14b72bea365b04ad521bd already in progress: creating tablet
I20240507 08:11:55.003201 5839 ts_tablet_manager.cc:1398] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240507 08:11:55.003502 5839 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.006395 5839 tablet_bootstrap.cc:654] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.008486 5940 server_base.cc:1034] running on GCE node
I20240507 08:11:55.009950 5940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240507 08:11:55.010267 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 400d2a7b0e224e46a5a580ca22e96eab from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.012164 5940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:11:55.013463 5940 hybrid_clock.cc:648] HybridClock initialized: now 1715069515013351 us; error 57 us; skew 500 ppm
I20240507 08:11:55.014357 5940 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:11:55.015286 5840 tablet_bootstrap.cc:654] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.018730 5940 webserver.cc:460] Webserver started at http://127.4.246.253:40743/ using document root <none> and password file <none>
I20240507 08:11:55.020483 5940 fs_manager.cc:359] Metadata directory not provided
I20240507 08:11:55.020762 5940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:11:55.022364 5843 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.022531 5843 ts_tablet_manager.cc:1367] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240507 08:11:55.023528 6100 tablet.cc:1722] T 6616660bc4a6400a9eb644590ced0e21 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:11:55.023694 5843 raft_consensus.cc:348] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.023972 5843 raft_consensus.cc:374] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.024078 5843 raft_consensus.cc:729] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.024298 6099 tablet.cc:1722] T 6616660bc4a6400a9eb644590ced0e21 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:11:55.034122 5839 tablet_bootstrap.cc:492] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.034111 5940 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.006s sys 0.000s
I20240507 08:11:55.034531 5839 ts_tablet_manager.cc:1367] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
W20240507 08:11:55.035214 5250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20240507 08:11:55.035470 5839 raft_consensus.cc:348] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.035676 5839 raft_consensus.cc:374] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.035759 5839 raft_consensus.cc:729] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.036005 5839 consensus_queue.cc:260] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.036244 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c1bde6a097f946a1bae0b57150f3aef5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.037256 5839 ts_tablet_manager.cc:1398] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:55.037811 5258 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c] alter complete (version 1)
W20240507 08:11:55.038538 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1bde6a097f946a1bae0b57150f3aef5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.039469 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet c9b8044e4a0a41218f67513b14c0ebeb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.039728 5838 tablet_replica.cc:330] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.039937 5838 raft_consensus.cc:2229] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.040099 5838 raft_consensus.cc:2258] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.041621 5838 ts_tablet_manager.cc:1875] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.044090 5843 consensus_queue.cc:260] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.044730 5843 ts_tablet_manager.cc:1398] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240507 08:11:55.045027 5843 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.045395 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16155f6d29ca4744902ede602ec6a8f3" } new_table_name: "default.Table_82367ef4edaa439dade10ab870371901" modify_external_catalogs: true
W20240507 08:11:55.045915 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240507 08:11:55.046769 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e49d8e80075e42cead2c50333bf2c4d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.047281 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e49d8e80075e42cead2c50333bf2c4d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.047436 5840 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.047610 5840 ts_tablet_manager.cc:1367] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.002s
I20240507 08:11:55.048311 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 87e1f43426b14f98b985656f8d0d7141 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.048497 5840 raft_consensus.cc:348] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.048736 5840 raft_consensus.cc:374] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.048821 5840 raft_consensus.cc:729] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.049105 5840 consensus_queue.cc:260] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.050756 5840 ts_tablet_manager.cc:1398] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:55.051142 5840 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.051394 5839 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.053038 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_67e5969a8d434e50bcd09e39a824f5c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.053339 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.053818 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67e5969a8d434e50bcd09e39a824f5c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:55.053870 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.055965 5841 ts_tablet_manager.cc:1888] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:55.059756 5843 tablet_bootstrap.cc:654] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.060595 5841 log.cc:1196] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/44f6f62c56cb4c29bf05914ca461a559
I20240507 08:11:55.061127 5841 ts_tablet_manager.cc:1909] T 44f6f62c56cb4c29bf05914ca461a559 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.061551 5839 tablet_bootstrap.cc:654] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.063896 5840 tablet_bootstrap.cc:654] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.066987 5838 ts_tablet_manager.cc:1888] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.069296 5838 log.cc:1196] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c9b8044e4a0a41218f67513b14c0ebeb
I20240507 08:11:55.069983 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 133) took 3550 ms. Trace:
I20240507 08:11:55.070168 5838 ts_tablet_manager.cc:1909] T c9b8044e4a0a41218f67513b14c0ebeb P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.070230 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_38102920150a4a08a922cdb41801b972" } new_table_name: "default.Table_307c9baefe434bb68c8d0906a9b9788e" modify_external_catalogs: true
I20240507 08:11:55.070137 5841 rpcz_store.cc:276] 0507 08:11:51.519964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.520143 (+ 179us) service_pool.cc:225] Handling call
0507 08:11:54.963391 (+3443248us) ts_tablet_manager.cc:1120] Deleting tablet 44f6f62c56cb4c29bf05914ca461a559
0507 08:11:54.963405 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.018839 (+ 55434us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.055811 (+ 36972us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.069922 (+ 14111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3443041}
I20240507 08:11:55.070670 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 44f6f62c56cb4c29bf05914ca461a559 (table default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]) successfully deleted
I20240507 08:11:55.070811 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 136) took 3429 ms. Trace:
I20240507 08:11:55.070911 5841 rpcz_store.cc:276] 0507 08:11:51.641355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.641437 (+ 82us) service_pool.cc:225] Handling call
0507 08:11:55.070645 (+3429208us) ts_tablet_manager.cc:1120] Deleting tablet 344bf4ab39f74accb383d695578de604
0507 08:11:55.070653 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.070758 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428924}
I20240507 08:11:55.071162 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c9b8044e4a0a41218f67513b14c0ebeb (table default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]) successfully deleted
W20240507 08:11:55.071318 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 344bf4ab39f74accb383d695578de604 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 344bf4ab39f74accb383d695578de604 already in progress: creating tablet
I20240507 08:11:55.071447 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 137) took 3412 ms. Trace:
W20240507 08:11:55.071498 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 645ae26bb26f43d298afe98817d88fa4 has no consensus state
I20240507 08:11:55.071554 5841 rpcz_store.cc:276] 0507 08:11:51.658895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.659388 (+ 493us) service_pool.cc:225] Handling call
0507 08:11:55.071273 (+3411885us) ts_tablet_manager.cc:1120] Deleting tablet d9ff5e1b6a9a476c8295a0d0ee85aecc
0507 08:11:55.071282 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.071383 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411338}
I20240507 08:11:55.072057 5841 tablet_replica.cc:330] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.072346 5841 raft_consensus.cc:2229] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:55.071820 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3edbc94e25c549b2b444e45bac4bc8e7" } new_table_name: "default.Table_1c799c3ed21d4c66bef27973c9e4688d" modify_external_catalogs: true
I20240507 08:11:55.072489 5838 tablet_replica.cc:330] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:11:55.072486 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d9ff5e1b6a9a476c8295a0d0ee85aecc already in progress: creating tablet
I20240507 08:11:55.072666 5841 pending_rounds.cc:62] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:55.072640 5838 raft_consensus.cc:2229] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.072721 5841 pending_rounds.cc:68] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924724823719936 op_type: NO_OP noop_request { }
I20240507 08:11:55.072821 5838 raft_consensus.cc:2258] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.072872 5841 raft_consensus.cc:2866] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:55.072966 5841 raft_consensus.cc:2258] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.073700 5841 ts_tablet_manager.cc:1875] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.074517 5838 ts_tablet_manager.cc:1875] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.076272 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3740c610e0f047f4b335d6e49868a581"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.076762 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3740c610e0f047f4b335d6e49868a581 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.077898 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7a752285620845dda2c19215e2ce252e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.078332 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a752285620845dda2c19215e2ce252e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.082772 5256 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:55.083863 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f9c3b5637d424b1199065768db5c961a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.084265 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f9c3b5637d424b1199065768db5c961a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.086040 5839 tablet_bootstrap.cc:492] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.086041 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.086308 5837 tablet_replica.cc:330] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.086491 5837 raft_consensus.cc:2229] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.086520 5839 ts_tablet_manager.cc:1367] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.004s
I20240507 08:11:55.086671 5837 raft_consensus.cc:2258] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.087591 5837 ts_tablet_manager.cc:1875] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.091070 5839 raft_consensus.cc:348] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.091291 5839 raft_consensus.cc:374] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.091403 5839 raft_consensus.cc:729] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.092036 5839 consensus_queue.cc:260] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.095029 5837 ts_tablet_manager.cc:1888] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.095165 5837 log.cc:1196] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/97a43c27b30a4c3cbfd80dce5f28d7c7
I20240507 08:11:55.095582 5837 ts_tablet_manager.cc:1909] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.095814 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5f853c15bb1145d8914412511c5172c1" } new_table_name: "default.Table_cd5c7f9c656d4df496741e31c1d5b370" modify_external_catalogs: true
I20240507 08:11:55.096177 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_13e0fb6057f044939108be222e8cb0f7" } new_table_name: "default.Table_4a0c1ff46db94e2791a9c86887b20998" modify_external_catalogs: true
I20240507 08:11:55.096877 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
I20240507 08:11:55.096927 5840 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.097064 5840 ts_tablet_manager.cc:1367] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
W20240507 08:11:55.098662 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b27cc38d0c4c4083900cd6765a5cf6e6 has no consensus state
W20240507 08:11:55.099493 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1fadbfd330b4e699e151f83125da281 has no consensus state
I20240507 08:11:55.099668 5840 raft_consensus.cc:348] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.099886 5840 raft_consensus.cc:374] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.099977 5840 raft_consensus.cc:729] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.100812 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.100979 5840 consensus_queue.cc:260] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.104314 5840 ts_tablet_manager.cc:1398] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:11:55.104333 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 400d2a7b0e224e46a5a580ca22e96eab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.104712 5840 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.107684 6118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:11:55.108911 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d0b3de590398468abb0a9e5277b608c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.109038 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 (DEFAULT_TABLE table=default.Table_ac1bf080fe40499889f843cc9bf7e236 [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.109061 5683 tablet_service.cc:1467] Processing CreateTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 (DEFAULT_TABLE table=default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.109134 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c" } modify_external_catalogs: true
I20240507 08:11:55.109316 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c" } modify_external_catalogs: true
W20240507 08:11:55.109547 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0b3de590398468abb0a9e5277b608c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.110410 5549 tablet_service.cc:1467] Processing CreateTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 (DEFAULT_TABLE table=default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.108911 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2335b4831b2749b5be46236a3f88c0e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.111753 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 (DEFAULT_TABLE table=default.Table_ac1bf080fe40499889f843cc9bf7e236 [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:55.111920 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2335b4831b2749b5be46236a3f88c0e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.112975 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27cc38d0c4c4083900cd6765a5cf6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.113814 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 31251576beee4a8a81a453b721c9c058
I20240507 08:11:55.113876 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43644b4405f44bfd9a1afcaedbd68db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.115098 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:11:55.115768 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4de7053cfaea4a54a402c9d826613c05
I20240507 08:11:55.116006 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 957775c01be94cc081db344c95a81310 (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.118096 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.118515 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3a7027841ed1443096999162f8cdd1ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.118749 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.118911 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.118989 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:55.119006 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a7027841ed1443096999162f8cdd1ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.119318 5839 ts_tablet_manager.cc:1398] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240507 08:11:55.120357 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.120532 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.121443 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27cc38d0c4c4083900cd6765a5cf6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.123044 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 (DEFAULT_TABLE table=default.Table_ac1bf080fe40499889f843cc9bf7e236 [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.124053 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43644b4405f44bfd9a1afcaedbd68db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.126252 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957775c01be94cc081db344c95a81310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.126768 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 (DEFAULT_TABLE table=default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.126868 5839 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.127152 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5471c74c33cd42e8aaf47f71f61adff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.129839 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 2754fc46c36346f9828198e3bb6ed3a7 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.130180 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2754fc46c36346f9828198e3bb6ed3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.130714 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 2754fc46c36346f9828198e3bb6ed3a7 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.132618 5422 tablet_service.cc:1467] Processing CreateTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 (DEFAULT_TABLE table=default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.132921 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43644b4405f44bfd9a1afcaedbd68db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.133955 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 957775c01be94cc081db344c95a81310 (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.134356 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27cc38d0c4c4083900cd6765a5cf6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.135126 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 (DEFAULT_TABLE table=default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:55.137224 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0b9f85cd78748d3afce83b64b3f823b consensus state has no leader
W20240507 08:11:55.137379 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d617213b959452ab6519809e56db128 consensus state has no leader
I20240507 08:11:55.137360 5940 fs_manager.cc:725] Time spent opening block manager: real 0.101s user 0.014s sys 0.004s
I20240507 08:11:55.137563 5940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:11:55.140933 5940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240507 08:11:55.143719 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 957775c01be94cc081db344c95a81310 (DEFAULT_TABLE table=default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.144219 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957775c01be94cc081db344c95a81310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.147166 5837 tablet_replica.cc:330] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.147436 5837 raft_consensus.cc:2229] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.147657 5837 raft_consensus.cc:2258] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.148517 5837 ts_tablet_manager.cc:1875] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.148934 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.149742 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2754fc46c36346f9828198e3bb6ed3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.149884 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 (DEFAULT_TABLE table=default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.150226 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.150506 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.150655 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 957775c01be94cc081db344c95a81310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.152021 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5471c74c33cd42e8aaf47f71f61adff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.155728 5840 tablet_bootstrap.cc:654] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.157812 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5471c74c33cd42e8aaf47f71f61adff7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.158286 5838 ts_tablet_manager.cc:1888] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.159524 5839 tablet_bootstrap.cc:654] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.162199 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 2754fc46c36346f9828198e3bb6ed3a7 (DEFAULT_TABLE table=default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.162758 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2754fc46c36346f9828198e3bb6ed3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.164104 5838 log.cc:1196] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/87e1f43426b14f98b985656f8d0d7141
I20240507 08:11:55.165627 5995 raft_consensus.cc:482] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.165782 5995 raft_consensus.cc:504] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.167230 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3f56ea5b7a43e4a89845da5a4d98f7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:55.167470 5718 raft_consensus.cc:2454] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:55.167902 5377 leader_election.cc:302] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:55.168280 6033 raft_consensus.cc:2781] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:55.168407 6033 raft_consensus.cc:482] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:55.168509 6033 raft_consensus.cc:3037] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.168931 5995 leader_election.cc:288] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.170452 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3f56ea5b7a43e4a89845da5a4d98f7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:55.170977 5378 leader_election.cc:341] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.171057 5837 ts_tablet_manager.cc:1888] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.171178 5837 log.cc:1196] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5cab176c4bd34f38b14ac096f79b912e
I20240507 08:11:55.171532 5837 ts_tablet_manager.cc:1909] T 5cab176c4bd34f38b14ac096f79b912e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.175175 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5cab176c4bd34f38b14ac096f79b912e (table default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]) successfully deleted
I20240507 08:11:55.177371 5840 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.177713 5840 ts_tablet_manager.cc:1367] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.000s
I20240507 08:11:55.178558 5840 raft_consensus.cc:348] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.178836 5840 raft_consensus.cc:374] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.178944 5840 raft_consensus.cc:729] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.179286 5840 consensus_queue.cc:260] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.180891 5840 ts_tablet_manager.cc:1398] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:11:55.181243 5840 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:55.183148 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 31251576beee4a8a81a453b721c9c058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31251576beee4a8a81a453b721c9c058 already in progress: creating tablet
I20240507 08:11:55.183802 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 5ca8fc2c7f3849c380fea56862ffc26e from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.184666 5840 tablet_bootstrap.cc:654] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:55.185951 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4de7053cfaea4a54a402c9d826613c05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de7053cfaea4a54a402c9d826613c05 already in progress: creating tablet
W20240507 08:11:55.186599 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3b5e3d23bbb6439a87b303dff1b44a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b5e3d23bbb6439a87b303dff1b44a1d already in progress: creating tablet
I20240507 08:11:55.187372 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.187848 5837 tablet_replica.cc:330] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.188081 5838 ts_tablet_manager.cc:1909] T 87e1f43426b14f98b985656f8d0d7141 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.188437 5837 raft_consensus.cc:2229] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.190905 5995 raft_consensus.cc:482] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.191463 5838 tablet_replica.cc:330] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.191757 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 87e1f43426b14f98b985656f8d0d7141 (table default.Table_9d3d56b2117448afac241847a046a08a [id=9ae1c0eb6fc4403b84c33e29d149a4df]) successfully deleted
I20240507 08:11:55.192162 5837 raft_consensus.cc:2258] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.192636 5838 raft_consensus.cc:2229] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.192883 5838 raft_consensus.cc:2258] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.193632 5837 ts_tablet_manager.cc:1875] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.196992 5995 raft_consensus.cc:504] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.197710 5995 leader_election.cc:288] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.198026 5838 ts_tablet_manager.cc:1875] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.199075 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6a35991c18414b9bc714e9a0ba14d8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.207345 5840 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.207546 5840 ts_tablet_manager.cc:1367] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
W20240507 08:11:55.208477 5377 leader_election.cc:341] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:55.210579 5843 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:11:55.211119 5378 leader_election.cc:341] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.211333 5378 leader_election.cc:302] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.212705 5995 raft_consensus.cc:2726] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.212996 5840 raft_consensus.cc:348] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.213546 5840 raft_consensus.cc:374] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.213719 5840 raft_consensus.cc:729] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.213696 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 7b4b84ab3f274185a2b838ebb969ab52 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.214017 5840 consensus_queue.cc:260] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.215626 5840 ts_tablet_manager.cc:1398] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:11:55.215997 5840 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.219128 5843 ts_tablet_manager.cc:1367] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.166s user 0.003s sys 0.003s
I20240507 08:11:55.220077 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet add94bcf0c844cb78059b7240c7e28c0 with delete_type TABLET_DATA_DELETED (replaced by tablet b7947ca2d8324dbb9c909ec50852fed8) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.220278 5843 raft_consensus.cc:348] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.220535 5843 raft_consensus.cc:374] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.220618 5843 raft_consensus.cc:729] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.221050 5843 consensus_queue.cc:260] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.226445 6033 raft_consensus.cc:504] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.231863 6033 leader_election.cc:288] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.233875 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3f56ea5b7a43e4a89845da5a4d98f7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:55.234210 5718 raft_consensus.cc:3037] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.238297 5840 tablet_bootstrap.cc:654] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.238442 5718 raft_consensus.cc:2454] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:55.240593 5837 ts_tablet_manager.cc:1888] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.240769 5837 log.cc:1196] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/ebc6cc72bb7e4721a8ab559102273db5
I20240507 08:11:55.241210 5837 ts_tablet_manager.cc:1909] T ebc6cc72bb7e4721a8ab559102273db5 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.242111 5377 leader_election.cc:302] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:55.242542 6033 raft_consensus.cc:2781] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:55.242880 6033 raft_consensus.cc:686] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:55.243245 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a3f56ea5b7a43e4a89845da5a4d98f7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:55.243527 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet ebc6cc72bb7e4721a8ab559102273db5 (table default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]) successfully deleted
I20240507 08:11:55.243883 5837 tablet_replica.cc:330] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.244093 5837 raft_consensus.cc:2229] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.244248 5837 raft_consensus.cc:2258] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.244957 5837 ts_tablet_manager.cc:1875] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.245499 6033 consensus_queue.cc:237] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.246338 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:11:55.247174 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:11:55.247207 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.248112 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:55.248490 5378 leader_election.cc:341] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.249513 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.251385 5840 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.251610 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.251636 5840 ts_tablet_manager.cc:1367] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.001s
I20240507 08:11:55.252636 5840 raft_consensus.cc:348] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.252946 5840 raft_consensus.cc:374] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.253048 5843 ts_tablet_manager.cc:1398] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240507 08:11:55.253100 5840 raft_consensus.cc:729] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.253520 5840 consensus_queue.cc:260] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.254554 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.257750 5840 ts_tablet_manager.cc:1398] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:55.259323 5840 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.261685 5840 tablet_bootstrap.cc:654] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.263892 5843 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.265013 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.266388 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.268545 5839 tablet_bootstrap.cc:492] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.269596 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet c4899c088531480ea15364af94595ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.269654 5839 ts_tablet_manager.cc:1367] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.149s user 0.019s sys 0.018s
I20240507 08:11:55.271245 5839 raft_consensus.cc:348] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.271883 5839 raft_consensus.cc:374] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.272002 5839 raft_consensus.cc:729] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.273435 5843 tablet_bootstrap.cc:654] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.274129 5839 consensus_queue.cc:260] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.274873 5839 ts_tablet_manager.cc:1398] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240507 08:11:55.276645 5839 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.277737 5840 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.277915 5840 ts_tablet_manager.cc:1367] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.002s
I20240507 08:11:55.278472 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5f94711838f74fb88cbb91fe4e7ff11b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.278846 5840 raft_consensus.cc:348] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:55.279103 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f94711838f74fb88cbb91fe4e7ff11b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.279268 5840 raft_consensus.cc:374] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.279451 5840 raft_consensus.cc:729] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.279759 5840 consensus_queue.cc:260] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.280411 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4994af241b3b4efe9518adcaa62fb422" } new_table_name: "default.Table_b1add7221be940138ee5b9e2e796e92c" modify_external_catalogs: true
I20240507 08:11:55.281459 5840 ts_tablet_manager.cc:1398] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:55.281668 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e19ac67deb044b61b0e6194e8b54aae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.282172 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e19ac67deb044b61b0e6194e8b54aae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.282603 5840 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.283291 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5d4ba83817d04005be08381a5c9c15b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.283737 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d4ba83817d04005be08381a5c9c15b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.284893 5840 tablet_bootstrap.cc:654] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.288847 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.292748 5837 ts_tablet_manager.cc:1888] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.296202 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a9968c4db5a54a7db162505d22dd31dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.296545 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 34e4e33f52df432b8d043171d7232c2a with delete_type TABLET_DATA_DELETED (replaced by tablet 98aa4ec7b93943e5bf8f9758b3bb71e0) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:55.296741 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9968c4db5a54a7db162505d22dd31dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.300243 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b21b58e8d49943268e7491117d5d717e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.300789 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b21b58e8d49943268e7491117d5d717e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.301339 5839 tablet_bootstrap.cc:654] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.303800 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2c00afce140541f39ea5236ad43ba1f5" } new_table_name: "default.Table_f2eabb2165474aadb0528bf86cbc1718" modify_external_catalogs: true
I20240507 08:11:55.311450 5840 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.312863 5837 log.cc:1196] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/add94bcf0c844cb78059b7240c7e28c0
I20240507 08:11:55.313308 5837 ts_tablet_manager.cc:1909] T add94bcf0c844cb78059b7240c7e28c0 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.314294 5840 ts_tablet_manager.cc:1367] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20240507 08:11:55.314342 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet add94bcf0c844cb78059b7240c7e28c0 (table default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:55.315120 5840 raft_consensus.cc:348] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.315377 5840 raft_consensus.cc:374] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.315479 5840 raft_consensus.cc:729] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.315749 5840 consensus_queue.cc:260] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.322103 5840 ts_tablet_manager.cc:1398] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:11:55.324246 5840 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:55.327206 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5471c74c33cd42e8aaf47f71f61adff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5471c74c33cd42e8aaf47f71f61adff7 already in progress: creating tablet
W20240507 08:11:55.327773 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e48a2b4a0e4088ad009a7f8fe77f3f already in progress: creating tablet
I20240507 08:11:55.329607 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7b4b84ab3f274185a2b838ebb969ab52
I20240507 08:11:55.329586 6087 raft_consensus.cc:482] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.329735 6087 raft_consensus.cc:504] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.330075 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 7b4b84ab3f274185a2b838ebb969ab52 deleted and replaced by tablet fed715ba993a4f0c9e54b209ef031981
I20240507 08:11:55.330497 6087 leader_election.cc:288] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:55.331228 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:55.331367 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.331710 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.331684 5837 tablet_replica.cc:330] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.331882 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.331997 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5ca8fc2c7f3849c380fea56862ffc26e
I20240507 08:11:55.332022 5837 raft_consensus.cc:2229] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.332250 5837 raft_consensus.cc:2258] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.332294 5656 leader_election.cc:341] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:55.332437 5654 leader_election.cc:341] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.332509 5255 catalog_manager.cc:6554] ReplaceTablet: tablet 5ca8fc2c7f3849c380fea56862ffc26e deleted and replaced by tablet 167ee9cbc9174b82824c468e098472a6
I20240507 08:11:55.332599 5840 tablet_bootstrap.cc:654] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.332566 5654 leader_election.cc:302] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.332875 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fc64369fe72742ccb59a630ee95a8007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.333443 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc64369fe72742ccb59a630ee95a8007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.333529 6087 raft_consensus.cc:2726] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.334045 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.334177 5841 ts_tablet_manager.cc:1888] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:55.334353 5841 log.cc:1196] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c9b8044e4a0a41218f67513b14c0ebeb
I20240507 08:11:55.334771 5837 ts_tablet_manager.cc:1875] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.334811 5841 ts_tablet_manager.cc:1909] T c9b8044e4a0a41218f67513b14c0ebeb P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.335328 5840 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.335466 5840 ts_tablet_manager.cc:1367] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.003s
I20240507 08:11:55.335675 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 139) took 3661 ms. Trace:
I20240507 08:11:55.335758 5841 rpcz_store.cc:276] 0507 08:11:51.674644 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.681618 (+ 6974us) service_pool.cc:225] Handling call
0507 08:11:55.071954 (+3390336us) ts_tablet_manager.cc:1120] Deleting tablet c9b8044e4a0a41218f67513b14c0ebeb
0507 08:11:55.071965 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.321605 (+249640us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.334051 (+ 12446us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.335617 (+ 1566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3390000}
I20240507 08:11:55.336266 5840 raft_consensus.cc:348] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.336570 5840 raft_consensus.cc:374] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.336654 5840 raft_consensus.cc:729] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.336881 5840 consensus_queue.cc:260] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.337334 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.339005 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.339298 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c9b8044e4a0a41218f67513b14c0ebeb (table default.Table_a51f156faabe436288474d53bdb248ad [id=e753ac224ae44c8fa51ba2e51a32c3a7]) successfully deleted
I20240507 08:11:55.341444 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.346907 5837 ts_tablet_manager.cc:1888] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.347051 5837 log.cc:1196] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c4899c088531480ea15364af94595ee5
I20240507 08:11:55.347434 5837 ts_tablet_manager.cc:1909] T c4899c088531480ea15364af94595ee5 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:11:55.351862 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c84a2018f1c442e2a4dfc1c6b9c5ceef consensus state has no leader
I20240507 08:11:55.352248 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_82367ef4edaa439dade10ab870371901" } new_table_name: "default.Table_c0775c295b804c7e987bafb2f8b19bed" modify_external_catalogs: true
I20240507 08:11:55.353770 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_21a782d34aee4be78034577651967f0b" } modify_external_catalogs: true
I20240507 08:11:55.353965 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_21a782d34aee4be78034577651967f0b" } modify_external_catalogs: true
I20240507 08:11:55.354727 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b570163e3d2740c1abb9a44675630cfc" } modify_external_catalogs: true
I20240507 08:11:55.354883 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b570163e3d2740c1abb9a44675630cfc" } modify_external_catalogs: true
I20240507 08:11:55.355381 5840 ts_tablet_manager.cc:1398] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240507 08:11:55.355620 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b8268009436d4ce088ecbc9aae6f3fdc" } modify_external_catalogs: true
I20240507 08:11:55.355676 5840 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.355763 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b8268009436d4ce088ecbc9aae6f3fdc" } modify_external_catalogs: true
I20240507 08:11:55.356534 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_871e9a698bf647929a1db0c34970c457"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.357256 5840 tablet_bootstrap.cc:654] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.357295 5838 ts_tablet_manager.cc:1888] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.357441 5838 log.cc:1196] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/400d2a7b0e224e46a5a580ca22e96eab
I20240507 08:11:55.357839 5838 ts_tablet_manager.cc:1909] T 400d2a7b0e224e46a5a580ca22e96eab P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.358743 5843 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.358892 5843 ts_tablet_manager.cc:1367] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.098s user 0.000s sys 0.012s
I20240507 08:11:55.359071 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 400d2a7b0e224e46a5a580ca22e96eab (table default.Table_8e33d7020c354b828f678dbbd57d5832 [id=079eefc78b9445f098838f484af7a3ad]) successfully deleted
W20240507 08:11:55.359145 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_871e9a698bf647929a1db0c34970c457 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:55.359380 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 31251576beee4a8a81a453b721c9c058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31251576beee4a8a81a453b721c9c058 already in progress: creating tablet
I20240507 08:11:55.359560 5839 tablet_bootstrap.cc:492] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.359704 5839 ts_tablet_manager.cc:1367] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.084s user 0.014s sys 0.005s
I20240507 08:11:55.359774 5843 raft_consensus.cc:348] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.358906 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c4899c088531480ea15364af94595ee5 (table default.Table_8ad438587c5c48c78d151533b16a55e3 [id=307b8b3612c1470d916ad0f37c986171]) successfully deleted
I20240507 08:11:55.359979 5843 raft_consensus.cc:374] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.360064 5843 raft_consensus.cc:729] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.360303 5843 consensus_queue.cc:260] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.360461 5839 raft_consensus.cc:348] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.361011 5839 raft_consensus.cc:374] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.361112 5839 raft_consensus.cc:729] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:11:55.361217 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f2de26190bc44916809856d84ad68345 has no consensus state
I20240507 08:11:55.361338 5839 consensus_queue.cc:260] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.361464 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet bde62374dc7f4a67a071b6b658e501c4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.362094 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_da387f5ea4c142428110959beb6cf669" } new_table_name: "default.Table_f517b84d78b54bc39275ee1d49b79bfd" modify_external_catalogs: true
I20240507 08:11:55.363332 5843 ts_tablet_manager.cc:1398] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:11:55.363848 5839 ts_tablet_manager.cc:1398] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:55.363631 5843 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.364120 5839 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.365979 5837 tablet_replica.cc:330] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.366202 5837 raft_consensus.cc:2229] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.366384 5837 raft_consensus.cc:2258] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.367506 5837 ts_tablet_manager.cc:1875] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.367982 5843 tablet_bootstrap.cc:654] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.373219 5840 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.373508 5840 ts_tablet_manager.cc:1367] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20240507 08:11:55.374462 5840 raft_consensus.cc:348] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.376431 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 141) took 3683 ms. Trace:
W20240507 08:11:55.377168 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3b5e3d23bbb6439a87b303dff1b44a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b5e3d23bbb6439a87b303dff1b44a1d already in progress: creating tablet
W20240507 08:11:55.377499 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4de7053cfaea4a54a402c9d826613c05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de7053cfaea4a54a402c9d826613c05 already in progress: creating tablet
I20240507 08:11:55.376036 5840 raft_consensus.cc:374] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.377758 5840 raft_consensus.cc:729] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.377991 5840 consensus_queue.cc:260] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.380259 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 44f6f62c56cb4c29bf05914ca461a559 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.381067 5840 ts_tablet_manager.cc:1398] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240507 08:11:55.377216 5841 rpcz_store.cc:276] 0507 08:11:51.692900 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.692960 (+ 60us) service_pool.cc:225] Handling call
0507 08:11:55.376234 (+3683274us) ts_tablet_manager.cc:1120] Deleting tablet f922a7519084409db98693ee8797592b
0507 08:11:55.376245 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.376360 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3683031}
I20240507 08:11:55.382958 5840 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:55.383092 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5471c74c33cd42e8aaf47f71f61adff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5471c74c33cd42e8aaf47f71f61adff7 already in progress: creating tablet
I20240507 08:11:55.383384 5837 ts_tablet_manager.cc:1888] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.383652 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 143) took 3677 ms. Trace:
W20240507 08:11:55.383710 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f922a7519084409db98693ee8797592b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f922a7519084409db98693ee8797592b already in progress: creating tablet
W20240507 08:11:55.384243 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e48a2b4a0e4088ad009a7f8fe77f3f already in progress: creating tablet
I20240507 08:11:55.384569 5838 tablet_replica.cc:330] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.384748 5838 raft_consensus.cc:2229] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.384876 5837 log.cc:1196] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5228590c8e7d4c34b3acab9eb16665d3
I20240507 08:11:55.384938 5838 raft_consensus.cc:2258] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.385435 5837 ts_tablet_manager.cc:1909] T 5228590c8e7d4c34b3acab9eb16665d3 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.385318 5841 rpcz_store.cc:276] 0507 08:11:51.706400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.709855 (+ 3455us) service_pool.cc:225] Handling call
0507 08:11:55.383499 (+3673644us) ts_tablet_manager.cc:1120] Deleting tablet ebc6cc72bb7e4721a8ab559102273db5
0507 08:11:55.383508 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.383604 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3673409}
I20240507 08:11:55.386265 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 166) took 3489 ms. Trace:
I20240507 08:11:55.386358 5841 rpcz_store.cc:276] 0507 08:11:51.897083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.897146 (+ 63us) service_pool.cc:225] Handling call
0507 08:11:55.386116 (+3488970us) ts_tablet_manager.cc:1120] Deleting tablet e49dfd218a24464fbac39a3f4b477462
0507 08:11:55.386126 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.386215 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3476219}
I20240507 08:11:55.386445 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5228590c8e7d4c34b3acab9eb16665d3 (table default.Table_eff2f49d93764317a595c308278c0732 [id=2b718529a9e7440d831c3d60a8c17ba4]) successfully deleted
I20240507 08:11:55.386703 5843 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.386853 5843 ts_tablet_manager.cc:1367] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.007s
I20240507 08:11:55.387379 5840 tablet_bootstrap.cc:654] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.387773 5843 raft_consensus.cc:348] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:55.388075 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 645ae26bb26f43d298afe98817d88fa4 has no consensus state
I20240507 08:11:55.388150 5843 raft_consensus.cc:374] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.388228 5843 raft_consensus.cc:729] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.388422 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e850547c67ff4a8fa952777edc77bd59" } new_table_name: "default.Table_54a33e3ca40248028587733beb4f6743" modify_external_catalogs: true
I20240507 08:11:55.388630 5843 consensus_queue.cc:260] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.385888 5839 tablet_bootstrap.cc:654] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:55.390172 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7b4b84ab3f274185a2b838ebb969ab52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b4b84ab3f274185a2b838ebb969ab52 already in progress: creating tablet
W20240507 08:11:55.390389 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5ca8fc2c7f3849c380fea56862ffc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ca8fc2c7f3849c380fea56862ffc26e already in progress: creating tablet
I20240507 08:11:55.386994 5838 ts_tablet_manager.cc:1875] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:55.391141 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ebc6cc72bb7e4721a8ab559102273db5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ebc6cc72bb7e4721a8ab559102273db5 already in progress: creating tablet
I20240507 08:11:55.391258 5843 ts_tablet_manager.cc:1398] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20240507 08:11:55.391394 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e49dfd218a24464fbac39a3f4b477462 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e49dfd218a24464fbac39a3f4b477462 already in progress: creating tablet
I20240507 08:11:55.391893 5843 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.392488 5840 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.392650 5840 ts_tablet_manager.cc:1367] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.001s
I20240507 08:11:55.392901 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 9e447562e82347a584e39586500c27c2 from {username='slave'} at 127.0.0.1:44738
W20240507 08:11:55.393393 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed2604a090c94c78bd4f7a1dea6f81b7 consensus state has no leader
I20240507 08:11:55.393502 5843 tablet_bootstrap.cc:654] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.393463 5840 raft_consensus.cc:348] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.393716 5840 raft_consensus.cc:374] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.393803 5840 raft_consensus.cc:729] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.394033 5840 consensus_queue.cc:260] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.394129 6073 raft_consensus.cc:482] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.394264 6073 raft_consensus.cc:504] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.394543 5840 ts_tablet_manager.cc:1398] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:55.394996 6073 leader_election.cc:288] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:55.396049 5517 leader_election.cc:341] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 44f6f62c56cb4c29bf05914ca461a559
W20240507 08:11:55.396368 5520 leader_election.cc:341] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 44f6f62c56cb4c29bf05914ca461a559
I20240507 08:11:55.396490 5520 leader_election.cc:302] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:55.398289 5838 ts_tablet_manager.cc:1888] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.398392 5838 log.cc:1196] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/34e4e33f52df432b8d043171d7232c2a
I20240507 08:11:55.398762 5838 ts_tablet_manager.cc:1909] T 34e4e33f52df432b8d043171d7232c2a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.398936 5840 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.399710 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 34e4e33f52df432b8d043171d7232c2a (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]) successfully deleted
I20240507 08:11:55.400490 6073 raft_consensus.cc:2726] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.400789 5840 tablet_bootstrap.cc:654] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.406136 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:55.409063 5840 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.409219 5843 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.409235 5840 ts_tablet_manager.cc:1367] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240507 08:11:55.409430 5843 ts_tablet_manager.cc:1367] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.001s
I20240507 08:11:55.409976 5840 raft_consensus.cc:348] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.410200 5840 raft_consensus.cc:374] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.410287 5840 raft_consensus.cc:729] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.410240 5843 raft_consensus.cc:348] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.410547 5843 raft_consensus.cc:374] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.410666 5843 raft_consensus.cc:729] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.410609 5840 consensus_queue.cc:260] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.410713 5258 catalog_manager.cc:5484] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:55.411187 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:55.411010 5843 consensus_queue.cc:260] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.413466 5840 ts_tablet_manager.cc:1398] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:55.415433 5843 ts_tablet_manager.cc:1398] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240507 08:11:55.417320 5840 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.418917 5843 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.419260 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:55.419389 5840 tablet_bootstrap.cc:654] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.420411 5841 tablet_replica.cc:330] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.420593 5841 raft_consensus.cc:2229] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.420821 5841 raft_consensus.cc:2258] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.421916 5843 tablet_bootstrap.cc:654] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.421991 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.422224 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.422457 5837 tablet_replica.cc:330] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:11:55.422619 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 consensus state has no leader
I20240507 08:11:55.422631 5837 raft_consensus.cc:2229] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.422786 5837 raft_consensus.cc:2258] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.422938 5841 ts_tablet_manager.cc:1875] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.423560 5837 ts_tablet_manager.cc:1875] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.426614 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 697611671ef64c53b0be4f5341b8f270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.428256 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 (DEFAULT_TABLE table=default.Table_f9c3b5637d424b1199065768db5c961a [id=866842c12bfe4b5aa14e08e8e3e226e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.428445 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 (DEFAULT_TABLE table=default.Table_f9c3b5637d424b1199065768db5c961a [id=866842c12bfe4b5aa14e08e8e3e226e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.428786 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb62a72ed2248f6886823b9c64ff4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.428876 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb62a72ed2248f6886823b9c64ff4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.430227 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 (DEFAULT_TABLE table=default.Table_f9c3b5637d424b1199065768db5c961a [id=866842c12bfe4b5aa14e08e8e3e226e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.430590 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb62a72ed2248f6886823b9c64ff4c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.430675 5547 tablet_service.cc:1467] Processing CreateTablet for tablet ba28c36f91e141f4a1372e7544b44281 (DEFAULT_TABLE table=default.Table_7a752285620845dda2c19215e2ce252e [id=a3381a4ec6114b93a1243df2b9ed02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.430681 5699 tablet_service.cc:1467] Processing CreateTablet for tablet ba28c36f91e141f4a1372e7544b44281 (DEFAULT_TABLE table=default.Table_7a752285620845dda2c19215e2ce252e [id=a3381a4ec6114b93a1243df2b9ed02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.431008 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28c36f91e141f4a1372e7544b44281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.431059 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28c36f91e141f4a1372e7544b44281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.434739 5413 tablet_service.cc:1467] Processing CreateTablet for tablet ba28c36f91e141f4a1372e7544b44281 (DEFAULT_TABLE table=default.Table_7a752285620845dda2c19215e2ce252e [id=a3381a4ec6114b93a1243df2b9ed02f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.435096 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28c36f91e141f4a1372e7544b44281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.438894 5698 tablet_service.cc:1467] Processing CreateTablet for tablet cd4379b6fded4a52843f55f27ae67121 (DEFAULT_TABLE table=default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.439317 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4379b6fded4a52843f55f27ae67121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.441210 5561 tablet_service.cc:1467] Processing CreateTablet for tablet cd4379b6fded4a52843f55f27ae67121 (DEFAULT_TABLE table=default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.441566 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4379b6fded4a52843f55f27ae67121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.443168 5425 tablet_service.cc:1467] Processing CreateTablet for tablet cd4379b6fded4a52843f55f27ae67121 (DEFAULT_TABLE table=default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.443588 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4379b6fded4a52843f55f27ae67121. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.446740 5841 ts_tablet_manager.cc:1888] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.446863 5841 log.cc:1196] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5228590c8e7d4c34b3acab9eb16665d3
I20240507 08:11:55.447286 5841 ts_tablet_manager.cc:1909] T 5228590c8e7d4c34b3acab9eb16665d3 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.448752 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_964368f3c7824528b36f3fb37bb2e215" } modify_external_catalogs: true
I20240507 08:11:55.448969 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_964368f3c7824528b36f3fb37bb2e215" } modify_external_catalogs: true
I20240507 08:11:55.450148 5837 ts_tablet_manager.cc:1888] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.450265 5837 log.cc:1196] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f29568ebe4774662a8b49b7829cf30e7
I20240507 08:11:55.450534 5843 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.450634 5837 ts_tablet_manager.cc:1909] T f29568ebe4774662a8b49b7829cf30e7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.450695 5843 ts_tablet_manager.cc:1367] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240507 08:11:55.451179 6087 raft_consensus.cc:482] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.451318 6087 raft_consensus.cc:504] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.451400 5843 raft_consensus.cc:348] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.451608 5843 raft_consensus.cc:374] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.451692 5843 raft_consensus.cc:729] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.451987 6087 leader_election.cc:288] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:55.452522 5843 consensus_queue.cc:260] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.452838 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.452936 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_54a33e3ca40248028587733beb4f6743" } modify_external_catalogs: true
I20240507 08:11:55.453136 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_54a33e3ca40248028587733beb4f6743" } modify_external_catalogs: true
W20240507 08:11:55.453414 5654 leader_election.cc:341] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.454485 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_88249527af574ff4a20c9e826c4be077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.454632 5837 tablet_replica.cc:330] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.454809 5837 raft_consensus.cc:2229] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.454948 5837 raft_consensus.cc:2258] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.455011 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88249527af574ff4a20c9e826c4be077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.455752 5837 ts_tablet_manager.cc:1875] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.456065 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:55.456573 5843 ts_tablet_manager.cc:1398] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20240507 08:11:55.456600 5656 leader_election.cc:341] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.456735 5656 leader_election.cc:302] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.457293 6087 raft_consensus.cc:2726] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.458374 5839 tablet_bootstrap.cc:492] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.458537 5839 ts_tablet_manager.cc:1367] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.001s
I20240507 08:11:55.459262 5839 raft_consensus.cc:348] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.459486 5839 raft_consensus.cc:374] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.459767 5843 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.459913 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 070fdbf0b00b4582a184729c2f2a4b69
I20240507 08:11:55.460709 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 168) took 3507 ms. Trace:
I20240507 08:11:55.460748 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.460065 5839 raft_consensus.cc:729] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.460753 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2603fc71c01940698e55cf91cc063fab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.460819 5841 rpcz_store.cc:276] 0507 08:11:51.953328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:51.953539 (+ 211us) service_pool.cc:225] Handling call
0507 08:11:55.416054 (+3462515us) ts_tablet_manager.cc:1120] Deleting tablet 5228590c8e7d4c34b3acab9eb16665d3
0507 08:11:55.420301 (+ 4247us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.426721 (+ 6420us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.446619 (+ 19898us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.460676 (+ 14057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3431581}
I20240507 08:11:55.461073 5839 consensus_queue.cc:260] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:55.461391 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2603fc71c01940698e55cf91cc063fab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.461831 5839 ts_tablet_manager.cc:1398] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:55.462558 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd
I20240507 08:11:55.462604 5839 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.462980 5262 catalog_manager.cc:6554] ReplaceTablet: tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd deleted and replaced by tablet c534708f1abd4a33965eea92bdc0d0de
I20240507 08:11:55.463852 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3f01e3311a8b4553a31914922fb08d13
I20240507 08:11:55.463845 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.463920 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 175) took 3432 ms. Trace:
I20240507 08:11:55.464026 5841 rpcz_store.cc:276] 0507 08:11:52.031913 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.031982 (+ 69us) service_pool.cc:225] Handling call
0507 08:11:55.463758 (+3431776us) ts_tablet_manager.cc:1120] Deleting tablet 60bc7bc06e494adcb6a8641b50b90e6c
0507 08:11:55.463792 (+ 34us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.463892 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3423670}
I20240507 08:11:55.464563 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9e447562e82347a584e39586500c27c2
I20240507 08:11:55.464607 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.464937 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 9e447562e82347a584e39586500c27c2 deleted and replaced by tablet 7d89e4185229431f9a68f599da4aff34
I20240507 08:11:55.465021 5843 tablet_bootstrap.cc:654] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.465116 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.465492 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5228590c8e7d4c34b3acab9eb16665d3 (table default.Table_eff2f49d93764317a595c308278c0732 [id=2b718529a9e7440d831c3d60a8c17ba4]) successfully deleted
W20240507 08:11:55.465737 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 60bc7bc06e494adcb6a8641b50b90e6c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60bc7bc06e494adcb6a8641b50b90e6c already in progress: creating tablet
I20240507 08:11:55.465771 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 178) took 3403 ms. Trace:
I20240507 08:11:55.465871 5841 rpcz_store.cc:276] 0507 08:11:52.062070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.062147 (+ 77us) service_pool.cc:225] Handling call
0507 08:11:55.465616 (+3403469us) ts_tablet_manager.cc:1120] Deleting tablet 270a1fd15925416483c211021549315c
0507 08:11:55.465626 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.465742 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3388618}
I20240507 08:11:55.466367 5837 ts_tablet_manager.cc:1888] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.466379 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 181) took 3382 ms. Trace:
I20240507 08:11:55.466497 5837 log.cc:1196] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:55.466477 5841 rpcz_store.cc:276] 0507 08:11:52.083436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.083492 (+ 56us) service_pool.cc:225] Handling call
0507 08:11:55.466265 (+3382773us) ts_tablet_manager.cc:1120] Deleting tablet 5554ebc621c24d459668f2cefc20b641
0507 08:11:55.466273 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.466351 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3377090}
I20240507 08:11:55.466869 5837 ts_tablet_manager.cc:1909] T e49dfd218a24464fbac39a3f4b477462 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.467096 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.467286 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.467427 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.467507 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.467593 5841 tablet_replica.cc:330] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.467777 5841 raft_consensus.cc:2229] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.467994 5841 raft_consensus.cc:2258] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.468508 5841 ts_tablet_manager.cc:1875] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:55.469204 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 270a1fd15925416483c211021549315c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 270a1fd15925416483c211021549315c already in progress: creating tablet
W20240507 08:11:55.469535 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5554ebc621c24d459668f2cefc20b641 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5554ebc621c24d459668f2cefc20b641 already in progress: creating tablet
I20240507 08:11:55.469844 6033 raft_consensus.cc:482] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.469995 6033 raft_consensus.cc:504] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.470590 6033 leader_election.cc:288] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.470819 5837 tablet_replica.cc:330] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.470885 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f29568ebe4774662a8b49b7829cf30e7 (table default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]) successfully deleted
I20240507 08:11:55.471002 5837 raft_consensus.cc:2229] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.471179 5837 raft_consensus.cc:2258] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.471367 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "812dd1ebd42f4e929838a0535c22ad69" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.471580 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.471796 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:55.471904 5378 leader_election.cc:341] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.472072 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.472329 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.473920 5840 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.474164 5840 ts_tablet_manager.cc:1367] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.004s
W20240507 08:11:55.474296 5377 leader_election.cc:341] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 812dd1ebd42f4e929838a0535c22ad69
I20240507 08:11:55.474453 5377 leader_election.cc:302] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.474926 6033 raft_consensus.cc:2726] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.475291 5840 raft_consensus.cc:348] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.475564 5840 raft_consensus.cc:374] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.475701 5840 raft_consensus.cc:729] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.475610 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6500147915384d518c459e558a9ce1d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.476083 5840 consensus_queue.cc:260] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:55.476238 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6500147915384d518c459e558a9ce1d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:55.476758 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5ca8fc2c7f3849c380fea56862ffc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ca8fc2c7f3849c380fea56862ffc26e already in progress: creating tablet
I20240507 08:11:55.474246 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 09adb78ad08a4e759eb9344cbabb6a9d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.472308 5839 tablet_bootstrap.cc:654] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.479941 5840 ts_tablet_manager.cc:1398] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:55.480814 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:11:55.481767 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.482403 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.483217 5840 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.484882 5840 tablet_bootstrap.cc:654] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:55.485610 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7b4b84ab3f274185a2b838ebb969ab52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b4b84ab3f274185a2b838ebb969ab52 already in progress: creating tablet
I20240507 08:11:55.485822 5837 ts_tablet_manager.cc:1875] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.486008 5838 tablet_replica.cc:330] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.486263 5838 raft_consensus.cc:2229] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.486442 5838 raft_consensus.cc:2258] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.487457 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e49dfd218a24464fbac39a3f4b477462 (table default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]) successfully deleted
W20240507 08:11:55.488158 5378 consensus_peers.cc:485] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:55.489807 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.490015 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d82625304942d38b0261f95e235a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.490773 5838 ts_tablet_manager.cc:1875] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.490936 5843 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.491094 5843 ts_tablet_manager.cc:1367] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240507 08:11:55.491854 5843 raft_consensus.cc:348] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.492087 5843 raft_consensus.cc:374] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.492180 5843 raft_consensus.cc:729] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.492411 5843 consensus_queue.cc:260] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:55.494623 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4306751d16444ec5948c2ba5831063f4 consensus state has no leader
I20240507 08:11:55.494721 5843 ts_tablet_manager.cc:1398] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:11:55.494709 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:11:55.495038 5843 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.496111 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.496675 5843 tablet_bootstrap.cc:654] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.498127 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.498373 5837 ts_tablet_manager.cc:1888] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.498502 5837 log.cc:1196] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/697611671ef64c53b0be4f5341b8f270
I20240507 08:11:55.498840 5837 ts_tablet_manager.cc:1909] T 697611671ef64c53b0be4f5341b8f270 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.500627 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.501322 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 697611671ef64c53b0be4f5341b8f270 (table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]) successfully deleted
I20240507 08:11:55.501641 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 973a199a533b4bbea3c2615acdbf95ca
W20240507 08:11:55.502084 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 070fdbf0b00b4582a184729c2f2a4b69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070fdbf0b00b4582a184729c2f2a4b69 already in progress: creating tablet
I20240507 08:11:55.502496 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.502652 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.503010 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:55.504760 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd already in progress: creating tablet
W20240507 08:11:55.505240 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3f01e3311a8b4553a31914922fb08d13 because the tablet was not found. No further retry: Not found: Tablet not found: 3f01e3311a8b4553a31914922fb08d13
I20240507 08:11:55.505415 5837 tablet_replica.cc:330] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:11:55.505605 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9e447562e82347a584e39586500c27c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e447562e82347a584e39586500c27c2 already in progress: creating tablet
I20240507 08:11:55.505656 5837 raft_consensus.cc:2229] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.505829 5837 raft_consensus.cc:2258] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.505995 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e1f9a56a95074e67bb33ceee1af48a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1f9a56a95074e67bb33ceee1af48a18 already in progress: creating tablet
I20240507 08:11:55.506615 5840 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.506788 5840 ts_tablet_manager.cc:1367] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.005s
I20240507 08:11:55.507489 5840 raft_consensus.cc:348] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.507712 5840 raft_consensus.cc:374] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.507793 5840 raft_consensus.cc:729] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.508090 5839 tablet_bootstrap.cc:492] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.508085 5841 ts_tablet_manager.cc:1888] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240507 08:11:55.508213 5841 log.cc:1196] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/cbf73c218b944679a12d769b1929e535
I20240507 08:11:55.508240 5839 ts_tablet_manager.cc:1367] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.001s
I20240507 08:11:55.508059 5840 consensus_queue.cc:260] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.509832 5841 ts_tablet_manager.cc:1909] T cbf73c218b944679a12d769b1929e535 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.510536 5839 raft_consensus.cc:348] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.510802 5839 raft_consensus.cc:374] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.510886 5840 ts_tablet_manager.cc:1398] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:11:55.510901 5839 raft_consensus.cc:729] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.511142 5839 consensus_queue.cc:260] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.511915 5839 ts_tablet_manager.cc:1398] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:11:55.514024 5838 ts_tablet_manager.cc:1888] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.514755 5839 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.514540 5840 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.516980 5839 tablet_bootstrap.cc:654] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.517714 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 179) took 3449 ms. Trace:
I20240507 08:11:55.518186 5837 ts_tablet_manager.cc:1875] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.518899 5840 tablet_bootstrap.cc:654] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.518754 5841 rpcz_store.cc:276] 0507 08:11:52.067968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.068066 (+ 98us) service_pool.cc:225] Handling call
0507 08:11:55.467513 (+3399447us) ts_tablet_manager.cc:1120] Deleting tablet cbf73c218b944679a12d769b1929e535
0507 08:11:55.467523 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.481435 (+ 13912us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.507946 (+ 26511us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.517655 (+ 9709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3364331}
I20240507 08:11:55.519694 5841 tablet_replica.cc:330] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.519827 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet cbf73c218b944679a12d769b1929e535 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]) successfully deleted
I20240507 08:11:55.520149 6033 raft_consensus.cc:482] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.520471 5841 raft_consensus.cc:2229] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.520401 6033 raft_consensus.cc:504] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.520644 5841 raft_consensus.cc:2258] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.521106 6033 leader_election.cc:288] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.521755 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405f264bac1464a8f1b4435715a64a4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:55.522504 5378 leader_election.cc:341] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.520294 5838 log.cc:1196] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/44f6f62c56cb4c29bf05914ca461a559
I20240507 08:11:55.523130 5838 ts_tablet_manager.cc:1909] T 44f6f62c56cb4c29bf05914ca461a559 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.524142 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 44f6f62c56cb4c29bf05914ca461a559 (table default.Table_f4619389ae4548d3add23a0eaf7bc6aa [id=88bd406fff2049d49bf6442b309f0c83]) successfully deleted
I20240507 08:11:55.524425 5843 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.524576 5843 ts_tablet_manager.cc:1367] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.004s
I20240507 08:11:55.524740 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet e1cf2df0d3bd46268e22713fcc135a74 with delete_type TABLET_DATA_DELETED (replaced by tablet f49dd4772f8e46478638b29c009c83ea) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.525264 5843 raft_consensus.cc:348] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.525476 5843 raft_consensus.cc:374] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.525552 5843 raft_consensus.cc:729] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.523344 5841 ts_tablet_manager.cc:1875] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:55.526176 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 070fdbf0b00b4582a184729c2f2a4b69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070fdbf0b00b4582a184729c2f2a4b69 already in progress: creating tablet
I20240507 08:11:55.528524 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.529281 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.530391 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 9ba24c078af9472da06793c58f147290 (table default.Table_70320eb4c56d4c38b5f30235746e7cb2 [id=8ea36271c6814ba58435556d3ff59f2a]). Expected version 1 got 0
W20240507 08:11:55.532450 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd already in progress: creating tablet
I20240507 08:11:55.533685 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405f264bac1464a8f1b4435715a64a4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:55.533660 5843 consensus_queue.cc:260] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.533918 5716 raft_consensus.cc:2454] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:55.536398 5839 tablet_bootstrap.cc:492] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.536582 5839 ts_tablet_manager.cc:1367] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240507 08:11:55.537417 5843 ts_tablet_manager.cc:1398] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240507 08:11:55.537613 5839 raft_consensus.cc:348] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.538614 5377 leader_election.cc:302] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.539237 6033 raft_consensus.cc:2781] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:55.539356 6033 raft_consensus.cc:482] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:55.539474 6033 raft_consensus.cc:3037] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.540442 5843 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.542222 5843 tablet_bootstrap.cc:654] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.543099 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:55.544428 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3f01e3311a8b4553a31914922fb08d13 because the tablet was not found. No further retry: Not found: Tablet not found: 3f01e3311a8b4553a31914922fb08d13
I20240507 08:11:55.544754 5841 ts_tablet_manager.cc:1888] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.544924 5841 log.cc:1196] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/09adb78ad08a4e759eb9344cbabb6a9d
W20240507 08:11:55.545326 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9e447562e82347a584e39586500c27c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e447562e82347a584e39586500c27c2 already in progress: creating tablet
I20240507 08:11:55.545384 5841 ts_tablet_manager.cc:1909] T 09adb78ad08a4e759eb9344cbabb6a9d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.545876 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 9ba24c078af9472da06793c58f147290 (table default.Table_70320eb4c56d4c38b5f30235746e7cb2 [id=8ea36271c6814ba58435556d3ff59f2a]). Expected version 1 got 0
I20240507 08:11:55.545884 6033 raft_consensus.cc:504] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.546267 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 184) took 3354 ms. Trace:
I20240507 08:11:55.546401 5841 rpcz_store.cc:276] 0507 08:11:52.191748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.191824 (+ 76us) service_pool.cc:225] Handling call
0507 08:11:55.519606 (+3327782us) ts_tablet_manager.cc:1120] Deleting tablet 09adb78ad08a4e759eb9344cbabb6a9d
0507 08:11:55.519616 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.529543 (+ 9927us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.544611 (+ 15068us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.546221 (+ 1610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3325948}
I20240507 08:11:55.546836 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 183) took 3359 ms. Trace:
I20240507 08:11:55.546937 5841 rpcz_store.cc:276] 0507 08:11:52.187712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.187780 (+ 68us) service_pool.cc:225] Handling call
0507 08:11:55.546686 (+3358906us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:55.546694 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.546807 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3352705}
I20240507 08:11:55.547397 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 185) took 3352 ms. Trace:
I20240507 08:11:55.547394 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 09adb78ad08a4e759eb9344cbabb6a9d (table default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]) successfully deleted
I20240507 08:11:55.547519 5841 rpcz_store.cc:276] 0507 08:11:52.194392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.194462 (+ 70us) service_pool.cc:225] Handling call
0507 08:11:55.547148 (+3352686us) ts_tablet_manager.cc:1120] Deleting tablet 5f2b43daab6b4deeac94fc6f1ff34a76
0507 08:11:55.547156 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.547229 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3350881}
W20240507 08:11:55.547778 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
W20240507 08:11:55.548023 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f2b43daab6b4deeac94fc6f1ff34a76 already in progress: creating tablet
I20240507 08:11:55.548105 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405f264bac1464a8f1b4435715a64a4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:55.548311 5716 raft_consensus.cc:3037] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.549372 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 187) took 3348 ms. Trace:
I20240507 08:11:55.549484 5841 rpcz_store.cc:276] 0507 08:11:52.200544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.200691 (+ 147us) service_pool.cc:225] Handling call
0507 08:11:55.549230 (+3348539us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:11:55.549239 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.549331 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3342630}
W20240507 08:11:55.549924 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5089db17ca904a04878d83209001c54e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5089db17ca904a04878d83209001c54e already in progress: creating tablet
I20240507 08:11:55.550709 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 186) took 3351 ms. Trace:
I20240507 08:11:55.550818 5841 rpcz_store.cc:276] 0507 08:11:52.198815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.198947 (+ 132us) service_pool.cc:225] Handling call
0507 08:11:55.550597 (+3351650us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:55.550605 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.550678 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3343658}
W20240507 08:11:55.551156 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:55.547086 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5405f264bac1464a8f1b4435715a64a4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:55.537845 5839 raft_consensus.cc:374] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:11:55.552022 5378 leader_election.cc:341] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.551697 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 188) took 3347 ms. Trace:
I20240507 08:11:55.552062 5839 raft_consensus.cc:729] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.552172 5841 rpcz_store.cc:276] 0507 08:11:52.204203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.239218 (+ 35015us) service_pool.cc:225] Handling call
0507 08:11:55.551582 (+3312364us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:11:55.551590 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.551668 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3194022}
I20240507 08:11:55.552224 5716 raft_consensus.cc:2454] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:11:55.552583 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 91ee3a6c4ac642a3921560ea13bc1341 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91ee3a6c4ac642a3921560ea13bc1341 already in progress: creating tablet
I20240507 08:11:55.552664 5377 leader_election.cc:302] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.553027 5841 tablet_replica.cc:330] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.553223 5841 raft_consensus.cc:2229] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.553390 5841 raft_consensus.cc:2258] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.553589 6033 leader_election.cc:288] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.553715 5995 raft_consensus.cc:2781] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:55.554673 5995 raft_consensus.cc:686] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:55.554966 5995 consensus_queue.cc:237] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.555400 5841 ts_tablet_manager.cc:1875] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.555326 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2335b4831b2749b5be46236a3f88c0e3" } new_table_name: "default.Table_c030c64baeaf4ec897fab752c5d13b23" modify_external_catalogs: true
W20240507 08:11:55.560457 5377 consensus_peers.cc:485] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d10b38120dfe4b9ab3c39cf584e4ff13. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:55.562588 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6b4f73d517f14c499174dbe4f4ede6c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.563304 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6b4f73d517f14c499174dbe4f4ede6c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.564740 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_63157d8c47004724ae6c531116bbf7dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.565356 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63157d8c47004724ae6c531116bbf7dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.552523 5839 consensus_queue.cc:260] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.568133 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_18d4760f5bf94ab69b1ce6418ad9507c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.568625 5839 ts_tablet_manager.cc:1398] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
W20240507 08:11:55.568683 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18d4760f5bf94ab69b1ce6418ad9507c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.570891 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cd5c7f9c656d4df496741e31c1d5b370" } new_table_name: "default.Table_97f2cc68bfaf4e558499da922503bdb1" modify_external_catalogs: true
I20240507 08:11:55.572322 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5efeda526693400790a6dfb0d118fac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.550117 5838 tablet_replica.cc:330] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:11:55.572795 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5efeda526693400790a6dfb0d118fac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.573093 5838 raft_consensus.cc:2229] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.574258 5838 raft_consensus.cc:2258] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.576844 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0542325c80241fca7e317cda64302a9 has no consensus state
I20240507 08:11:55.577543 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_703177a8d58346c88798c7fef25a5930"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.578171 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_703177a8d58346c88798c7fef25a5930 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.580036 5841 ts_tablet_manager.cc:1888] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.581478 5838 ts_tablet_manager.cc:1875] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.581723 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4737c51589da4300bf37cc2638784041"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.582276 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4737c51589da4300bf37cc2638784041 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.583348 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 91c6cb7920df47aa97be6555c587798a from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.585076 5260 catalog_manager.cc:5484] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:55.587111 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 33875331045146a9a02559a0b37d5c6f (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.588353 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 33875331045146a9a02559a0b37d5c6f (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.588788 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33875331045146a9a02559a0b37d5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.588934 5839 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.591065 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33875331045146a9a02559a0b37d5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.592871 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 (DEFAULT_TABLE table=default.Table_e49d8e80075e42cead2c50333bf2c4d3 [id=4cd92a7dd38946788551d84e24279035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.593222 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831153d2b89e43f8b08b086f7a8f3a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.594550 5839 tablet_bootstrap.cc:654] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.608583 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 (DEFAULT_TABLE table=default.Table_e49d8e80075e42cead2c50333bf2c4d3 [id=4cd92a7dd38946788551d84e24279035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.609165 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831153d2b89e43f8b08b086f7a8f3a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.614532 5838 ts_tablet_manager.cc:1888] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.616616 5837 ts_tablet_manager.cc:1888] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.617835 5840 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.618059 5840 ts_tablet_manager.cc:1367] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.104s user 0.002s sys 0.003s
I20240507 08:11:55.618821 5840 raft_consensus.cc:348] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.619629 5840 raft_consensus.cc:374] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.619807 5840 raft_consensus.cc:729] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.620117 5840 consensus_queue.cc:260] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.620852 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 (DEFAULT_TABLE table=default.Table_be493cfc9faf43faa5b0a4afe7287d6d [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.621255 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dc58133d2b4f5496b4e570ae6494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.623299 5840 ts_tablet_manager.cc:1398] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240507 08:11:55.588418 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 33875331045146a9a02559a0b37d5c6f (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.624478 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33875331045146a9a02559a0b37d5c6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.625020 5843 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.625170 5843 ts_tablet_manager.cc:1367] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.007s
I20240507 08:11:55.610117 6155 raft_consensus.cc:482] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.615314 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 (DEFAULT_TABLE table=default.Table_e49d8e80075e42cead2c50333bf2c4d3 [id=4cd92a7dd38946788551d84e24279035]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.626081 5843 raft_consensus.cc:348] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.626246 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 831153d2b89e43f8b08b086f7a8f3a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.626279 5843 raft_consensus.cc:374] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.626375 5843 raft_consensus.cc:729] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.626633 5843 consensus_queue.cc:260] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.629243 5843 ts_tablet_manager.cc:1398] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:55.629300 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 (DEFAULT_TABLE table=default.Table_be493cfc9faf43faa5b0a4afe7287d6d [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.629524 5843 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.629694 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dc58133d2b4f5496b4e570ae6494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.630995 6155 raft_consensus.cc:504] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.631565 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 753019050fcf4dc3b69722c3eb475037 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.631726 5840 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.634282 5837 log.cc:1196] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/80d82625304942d38b0261f95e235a41
I20240507 08:11:55.634905 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:55.635139 5716 raft_consensus.cc:2454] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:55.635403 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c030c64baeaf4ec897fab752c5d13b23" } new_table_name: "default.Table_8c28748ca00b43e7bff97d436034daaf" modify_external_catalogs: true
I20240507 08:11:55.635668 5837 ts_tablet_manager.cc:1909] T 80d82625304942d38b0261f95e235a41 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.636772 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.637320 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.637437 5840 tablet_bootstrap.cc:654] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.615658 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 (DEFAULT_TABLE table=default.Table_be493cfc9faf43faa5b0a4afe7287d6d [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.638767 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85dc58133d2b4f5496b4e570ae6494c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.633150 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet f968f122c39b4fd5b995e8a2848a52d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:55.642614 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d617213b959452ab6519809e56db128 consensus state has no leader
I20240507 08:11:55.645381 5838 log.cc:1196] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/09adb78ad08a4e759eb9344cbabb6a9d
I20240507 08:11:55.645920 5838 ts_tablet_manager.cc:1909] T 09adb78ad08a4e759eb9344cbabb6a9d P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.647445 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet f5e48a2b4a0e4088ad009a7f8fe77f3f from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.647833 5838 tablet_replica.cc:330] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.647948 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_487b0259f6ea4d48add257b1e7997937" } new_table_name: "default.Table_610ce2d36eaf4152b4c056012a719c18" modify_external_catalogs: true
I20240507 08:11:55.648072 5838 raft_consensus.cc:2229] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.648247 5838 raft_consensus.cc:2258] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.648890 5377 leader_election.cc:302] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:55.648923 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.649510 5838 ts_tablet_manager.cc:1875] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.650058 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_88249527af574ff4a20c9e826c4be077" } new_table_name: "default.Table_75025136df1f453e92ffc67c475fbb56" modify_external_catalogs: true
I20240507 08:11:55.650542 5839 tablet_bootstrap.cc:492] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.650674 5839 ts_tablet_manager.cc:1367] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.029s
I20240507 08:11:55.651397 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 09adb78ad08a4e759eb9344cbabb6a9d (table default.Table_a477a636c30a4d289f6c00d2975f1e5c [id=f65c5fc893d147a3b9c1cb1d069f188b]) successfully deleted
W20240507 08:11:55.651630 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e1f9a56a95074e67bb33ceee1af48a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1f9a56a95074e67bb33ceee1af48a18 already in progress: creating tablet
W20240507 08:11:55.651859 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 because the tablet was not found. No further retry: Not found: Tablet not found: 84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:11:55.651866 5839 raft_consensus.cc:348] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.652045 5839 raft_consensus.cc:374] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:11:55.652051 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 973a199a533b4bbea3c2615acdbf95ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973a199a533b4bbea3c2615acdbf95ca already in progress: creating tablet
I20240507 08:11:55.652127 5839 raft_consensus.cc:729] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.652370 5839 consensus_queue.cc:260] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.653786 5839 ts_tablet_manager.cc:1398] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:55.654107 5839 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.655145 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 80d82625304942d38b0261f95e235a41 (table default.Table_014d10f31cfe4b05a42d8f9c3d3921e3 [id=478717ae383042c8848a113d90cc85ed]) successfully deleted
W20240507 08:11:55.655519 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 because the tablet was not found. No further retry: Not found: Tablet not found: 84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:11:55.655699 5839 tablet_bootstrap.cc:654] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:55.655884 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 973a199a533b4bbea3c2615acdbf95ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973a199a533b4bbea3c2615acdbf95ca already in progress: creating tablet
I20240507 08:11:55.657397 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_44cd8bf73a544ecf89f263225686b5fd [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 2 got 0
I20240507 08:11:55.660044 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.660825 5837 tablet_replica.cc:330] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.661157 5837 raft_consensus.cc:2229] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.661223 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_44cd8bf73a544ecf89f263225686b5fd [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 2 got 0
I20240507 08:11:55.661476 5837 raft_consensus.cc:2258] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.661917 6155 leader_election.cc:288] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:55.662441 5837 ts_tablet_manager.cc:1875] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.663604 5841 log.cc:1196] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f968f122c39b4fd5b995e8a2848a52d4
I20240507 08:11:55.664173 5841 ts_tablet_manager.cc:1909] T f968f122c39b4fd5b995e8a2848a52d4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.665210 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 194) took 3199 ms. Trace:
I20240507 08:11:55.665323 5841 rpcz_store.cc:276] 0507 08:11:52.466193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.466252 (+ 59us) service_pool.cc:225] Handling call
0507 08:11:55.552960 (+3086708us) ts_tablet_manager.cc:1120] Deleting tablet f968f122c39b4fd5b995e8a2848a52d4
0507 08:11:55.552968 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.561307 (+ 8339us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.579918 (+ 18611us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.665137 (+ 85219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3084180}
I20240507 08:11:55.665787 5841 tablet_replica.cc:330] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.665794 5555 tablet_service.cc:1467] Processing CreateTablet for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (DEFAULT_TABLE table=default.Table_d8b43861f60b4b6189e6386d097240a9 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.665979 5841 raft_consensus.cc:2229] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:55.666265 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefbe3a60f84cadb5fd3e69c6efdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.666311 5841 pending_rounds.cc:62] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:55.666384 5841 pending_rounds.cc:68] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924727257227264 op_type: NO_OP noop_request { }
I20240507 08:11:55.666455 5839 tablet_bootstrap.cc:492] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.666530 5841 raft_consensus.cc:2866] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:55.666618 5839 ts_tablet_manager.cc:1367] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20240507 08:11:55.666630 5841 raft_consensus.cc:2258] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.667192 5841 ts_tablet_manager.cc:1875] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.667577 5839 raft_consensus.cc:348] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.667797 5839 raft_consensus.cc:374] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.667883 5839 raft_consensus.cc:729] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.668156 5839 consensus_queue.cc:260] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:55.670166 5378 leader_election.cc:341] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.670801 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f968f122c39b4fd5b995e8a2848a52d4 (table default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]) successfully deleted
I20240507 08:11:55.671229 5839 ts_tablet_manager.cc:1398] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:11:55.671262 5838 ts_tablet_manager.cc:1888] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.671573 5839 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.673664 5837 ts_tablet_manager.cc:1888] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.674047 5995 raft_consensus.cc:2781] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:55.674165 5995 raft_consensus.cc:482] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:55.674463 5837 log.cc:1196] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/ea17889619fa420fbeacf1b41594c598
I20240507 08:11:55.675144 5837 ts_tablet_manager.cc:1909] T ea17889619fa420fbeacf1b41594c598 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.675298 5995 raft_consensus.cc:3037] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.677826 5839 tablet_bootstrap.cc:654] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.679158 5843 tablet_bootstrap.cc:654] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.684178 5995 raft_consensus.cc:504] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.685611 5425 tablet_service.cc:1467] Processing CreateTablet for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (DEFAULT_TABLE table=default.Table_d8b43861f60b4b6189e6386d097240a9 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.686057 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefbe3a60f84cadb5fd3e69c6efdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.686275 5995 leader_election.cc:288] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:55.686427 5995 raft_consensus.cc:482] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.686604 5995 raft_consensus.cc:504] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.686744 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:55.687403 5378 leader_election.cc:341] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.687801 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4a0c1ff46db94e2791a9c86887b20998" } modify_external_catalogs: true
W20240507 08:11:55.688021 5377 leader_election.cc:341] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5cab176c4bd34f38b14ac096f79b912e
I20240507 08:11:55.687999 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4a0c1ff46db94e2791a9c86887b20998" } modify_external_catalogs: true
I20240507 08:11:55.688176 5995 leader_election.cc:288] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.688133 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5cab176c4bd34f38b14ac096f79b912e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:55.688632 5378 leader_election.cc:341] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.688759 5378 leader_election.cc:302] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.689498 5995 raft_consensus.cc:2726] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.685652 5413 tablet_service.cc:1467] Processing CreateTablet for tablet b9e715a29b0643b6ac61f9e73874c25c (DEFAULT_TABLE table=default.Table_c2e5247aef1b421c8e538b0796249648 [id=4195f2f42cc744eda867dbfe5f8fd202]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.689955 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e715a29b0643b6ac61f9e73874c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.690025 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:55.691181 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70320eb4c56d4c38b5f30235746e7cb2" } modify_external_catalogs: true
I20240507 08:11:55.691483 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70320eb4c56d4c38b5f30235746e7cb2" } modify_external_catalogs: true
I20240507 08:11:55.692344 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ede2eb6eb3b247c2ac451571f36e8e29" } modify_external_catalogs: true
I20240507 08:11:55.692561 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ede2eb6eb3b247c2ac451571f36e8e29" } modify_external_catalogs: true
I20240507 08:11:55.698007 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 645ae26bb26f43d298afe98817d88fa4 (DEFAULT_TABLE table=default.Table_c0775c295b804c7e987bafb2f8b19bed [id=482211c676ea45a681f5a8bcd7cdb8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.698948 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645ae26bb26f43d298afe98817d88fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.700402 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 6616660bc4a6400a9eb644590ced0e21 (table default.Table_f485dedc2a6849ff8b388a9b07e06b73 [id=e5e2009b88a343018ab45566f5d2246c]). Expected version 1 got 0
I20240507 08:11:55.703433 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 645ae26bb26f43d298afe98817d88fa4 (DEFAULT_TABLE table=default.Table_c0775c295b804c7e987bafb2f8b19bed [id=482211c676ea45a681f5a8bcd7cdb8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.703831 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645ae26bb26f43d298afe98817d88fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.685665 5683 tablet_service.cc:1467] Processing CreateTablet for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (DEFAULT_TABLE table=default.Table_d8b43861f60b4b6189e6386d097240a9 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.685829 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cbf7dd6413fb47108cce25cde80738fe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:55.685482 5698 tablet_service.cc:1467] Processing CreateTablet for tablet b9e715a29b0643b6ac61f9e73874c25c (DEFAULT_TABLE table=default.Table_c2e5247aef1b421c8e538b0796249648 [id=4195f2f42cc744eda867dbfe5f8fd202]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.706820 5716 raft_consensus.cc:3037] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.707031 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e715a29b0643b6ac61f9e73874c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.703099 5839 tablet_bootstrap.cc:492] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.709738 5839 ts_tablet_manager.cc:1367] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.001s
I20240507 08:11:55.697983 5555 tablet_service.cc:1467] Processing CreateTablet for tablet b9e715a29b0643b6ac61f9e73874c25c (DEFAULT_TABLE table=default.Table_c2e5247aef1b421c8e538b0796249648 [id=4195f2f42cc744eda867dbfe5f8fd202]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.710256 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b9e715a29b0643b6ac61f9e73874c25c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.710644 5839 raft_consensus.cc:348] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.710881 5839 raft_consensus.cc:374] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.699430 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 645ae26bb26f43d298afe98817d88fa4 (DEFAULT_TABLE table=default.Table_c0775c295b804c7e987bafb2f8b19bed [id=482211c676ea45a681f5a8bcd7cdb8a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.711004 5839 raft_consensus.cc:729] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.713341 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30a40b16a51c4ad28c5fee73ed5e4185" } modify_external_catalogs: true
I20240507 08:11:55.708953 5838 log.cc:1196] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e1cf2df0d3bd46268e22713fcc135a74
I20240507 08:11:55.713543 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30a40b16a51c4ad28c5fee73ed5e4185" } modify_external_catalogs: true
I20240507 08:11:55.714012 5838 ts_tablet_manager.cc:1909] T e1cf2df0d3bd46268e22713fcc135a74 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.714996 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e1cf2df0d3bd46268e22713fcc135a74 (table default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]) successfully deleted
I20240507 08:11:55.715801 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 753019050fcf4dc3b69722c3eb475037
I20240507 08:11:55.716277 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 753019050fcf4dc3b69722c3eb475037 deleted and replaced by tablet aca8c38841db45e38ae5d8d965a16d0e
I20240507 08:11:55.716470 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.716740 5839 consensus_queue.cc:260] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.717887 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.718963 5841 ts_tablet_manager.cc:1888] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:55.719202 5841 log.cc:1196] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/34e4e33f52df432b8d043171d7232c2a
I20240507 08:11:55.719213 5838 tablet_replica.cc:330] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.719419 5838 raft_consensus.cc:2229] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.719615 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 5e53409fe13e4a9585dbba648a25cc07 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.720181 5841 ts_tablet_manager.cc:1909] T 34e4e33f52df432b8d043171d7232c2a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.720981 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 193) took 3261 ms. Trace:
I20240507 08:11:55.721086 5841 rpcz_store.cc:276] 0507 08:11:52.459428 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.459505 (+ 77us) service_pool.cc:225] Handling call
0507 08:11:55.665645 (+3206140us) ts_tablet_manager.cc:1120] Deleting tablet 34e4e33f52df432b8d043171d7232c2a
0507 08:11:55.665655 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.690409 (+ 24754us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.718857 (+ 28448us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.720923 (+ 2066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3192581}
I20240507 08:11:55.721632 5839 ts_tablet_manager.cc:1398] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:11:55.721628 5838 raft_consensus.cc:2258] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.722007 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_703177a8d58346c88798c7fef25a5930" } modify_external_catalogs: true
I20240507 08:11:55.722226 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_703177a8d58346c88798c7fef25a5930" } modify_external_catalogs: true
I20240507 08:11:55.724627 5838 ts_tablet_manager.cc:1875] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.725122 5843 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.725332 5843 ts_tablet_manager.cc:1367] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.096s user 0.008s sys 0.001s
I20240507 08:11:55.725442 5839 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.726125 5843 raft_consensus.cc:348] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.726401 5843 raft_consensus.cc:374] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.726490 5843 raft_consensus.cc:729] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.726780 5843 consensus_queue.cc:260] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.726915 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet ea17889619fa420fbeacf1b41594c598 (table default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]) successfully deleted
I20240507 08:11:55.727133 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.727269 5839 tablet_bootstrap.cc:654] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.727501 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet ebc6cc72bb7e4721a8ab559102273db5 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.729681 5843 ts_tablet_manager.cc:1398] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:55.730005 5843 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.731124 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 (DEFAULT_TABLE table=default.Table_4d38bf47908640beb3f6d94ad4446717 [id=84ee77d848974c2d8ccd5d9d518e1ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.731544 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767905763ba14cb8bbc52151ae9f17e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.732177 5843 tablet_bootstrap.cc:654] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.732736 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 195) took 3239 ms. Trace:
I20240507 08:11:55.732849 5841 rpcz_store.cc:276] 0507 08:11:52.482525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.482628 (+ 103us) service_pool.cc:225] Handling call
0507 08:11:55.721408 (+3238780us) ts_tablet_manager.cc:1120] Deleting tablet c33710c6b55744c4ba0843c228abded3
0507 08:11:55.721417 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.721515 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3236668}
I20240507 08:11:55.733649 5841 tablet_replica.cc:330] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.733827 5841 raft_consensus.cc:2229] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.734351 5841 raft_consensus.cc:2258] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.734601 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c33710c6b55744c4ba0843c228abded3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c33710c6b55744c4ba0843c228abded3 already in progress: creating tablet
I20240507 08:11:55.735743 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 (DEFAULT_TABLE table=default.Table_4d38bf47908640beb3f6d94ad4446717 [id=84ee77d848974c2d8ccd5d9d518e1ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.736371 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767905763ba14cb8bbc52151ae9f17e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:55.736552 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0542325c80241fca7e317cda64302a9 has no consensus state
I20240507 08:11:55.736855 5838 ts_tablet_manager.cc:1888] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.736980 5838 log.cc:1196] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f968f122c39b4fd5b995e8a2848a52d4
I20240507 08:11:55.737358 5838 ts_tablet_manager.cc:1909] T f968f122c39b4fd5b995e8a2848a52d4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.735745 5561 tablet_service.cc:1467] Processing CreateTablet for tablet f2de26190bc44916809856d84ad68345 (DEFAULT_TABLE table=default.Table_f2eabb2165474aadb0528bf86cbc1718 [id=ff9529d9a26e41cb9ebf472b849d6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.738098 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de26190bc44916809856d84ad68345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.738458 5838 tablet_replica.cc:330] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:55.738673 5838 raft_consensus.cc:2229] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.738896 5838 raft_consensus.cc:2258] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.741235 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f968f122c39b4fd5b995e8a2848a52d4 (table default.Table_8aaa2c0a5ef44d2bb4718bbc70104607 [id=1ca8359849f84041a3c9ad211ae57cf9]) successfully deleted
W20240507 08:11:55.741542 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 753019050fcf4dc3b69722c3eb475037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753019050fcf4dc3b69722c3eb475037 already in progress: creating tablet
I20240507 08:11:55.742323 5841 ts_tablet_manager.cc:1875] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.742846 5838 ts_tablet_manager.cc:1875] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.749637 5843 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.749802 5843 ts_tablet_manager.cc:1367] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240507 08:11:55.750500 5841 ts_tablet_manager.cc:1888] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:55.750679 5841 log.cc:1196] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/baa842552d00489998f83e68aa0d516b
I20240507 08:11:55.750775 5843 raft_consensus.cc:348] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.751019 5843 raft_consensus.cc:374] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.751118 5843 raft_consensus.cc:729] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.751384 5841 ts_tablet_manager.cc:1909] T baa842552d00489998f83e68aa0d516b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.751420 5843 consensus_queue.cc:260] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.752365 6073 raft_consensus.cc:482] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.753031 5843 ts_tablet_manager.cc:1398] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:55.753326 5843 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.754989 5843 tablet_bootstrap.cc:654] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.755924 6073 raft_consensus.cc:504] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.756793 6073 leader_election.cc:288] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:55.758394 5410 tablet_service.cc:1467] Processing CreateTablet for tablet f2de26190bc44916809856d84ad68345 (DEFAULT_TABLE table=default.Table_f2eabb2165474aadb0528bf86cbc1718 [id=ff9529d9a26e41cb9ebf472b849d6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.758781 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de26190bc44916809856d84ad68345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.760778 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 (DEFAULT_TABLE table=default.Table_54a33e3ca40248028587733beb4f6743 [id=9b06744f353d42a69ea7f5b5b60f3a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.761325 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.762411 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4cd7b56d4e468eb754d9653117d023 (DEFAULT_TABLE table=default.Table_c1bde6a097f946a1bae0b57150f3aef5 [id=aac50aefe6524a84b1e4564d3f0ce394]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.762804 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4cd7b56d4e468eb754d9653117d023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.762784 5838 ts_tablet_manager.cc:1888] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.762934 5838 log.cc:1196] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:55.763286 5838 ts_tablet_manager.cc:1909] T 0a6a35991c18414b9bc714e9a0ba14d8 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:55.764295 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 0a6a35991c18414b9bc714e9a0ba14d8 (table default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]) successfully deleted
I20240507 08:11:55.766270 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:55.766433 5843 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.766593 5843 ts_tablet_manager.cc:1367] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240507 08:11:55.767313 5843 raft_consensus.cc:348] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.767518 5843 raft_consensus.cc:374] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.767596 5843 raft_consensus.cc:729] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.767897 5843 consensus_queue.cc:260] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.768522 5843 ts_tablet_manager.cc:1398] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:55.768978 5843 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.770680 5843 tablet_bootstrap.cc:654] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.771916 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:55.774027 5839 tablet_bootstrap.cc:492] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.774220 5839 ts_tablet_manager.cc:1367] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.050s user 0.007s sys 0.001s
I20240507 08:11:55.774798 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a6a35991c18414b9bc714e9a0ba14d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.775588 5839 raft_consensus.cc:348] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.775815 5839 raft_consensus.cc:374] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.775902 5839 raft_consensus.cc:729] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.776130 5839 consensus_queue.cc:260] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.779724 5839 ts_tablet_manager.cc:1398] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:11:55.781526 5839 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.783242 5839 tablet_bootstrap.cc:654] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:55.784655 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0a6a35991c18414b9bc714e9a0ba14d8 because the tablet was not found. No further retry: Not found: Tablet not found: 0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:55.784721 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4cd7b56d4e468eb754d9653117d023 (DEFAULT_TABLE table=default.Table_c1bde6a097f946a1bae0b57150f3aef5 [id=aac50aefe6524a84b1e4564d3f0ce394]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.785195 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4cd7b56d4e468eb754d9653117d023. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:55.785737 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa094773f3bf418492bcd5b5021fd429 consensus state has no leader
I20240507 08:11:55.789902 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 209) took 2935 ms. Trace:
W20240507 08:11:55.789934 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6454c4a59d4c4cae1681bcae0da2d7 has no consensus state
I20240507 08:11:55.790128 5841 rpcz_store.cc:276] 0507 08:11:52.854786 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.854864 (+ 78us) service_pool.cc:225] Handling call
0507 08:11:55.733123 (+2878259us) ts_tablet_manager.cc:1120] Deleting tablet baa842552d00489998f83e68aa0d516b
0507 08:11:55.733132 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.745912 (+ 12780us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.750374 (+ 4462us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.789861 (+ 39487us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2858235}
I20240507 08:11:55.790556 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:11:55.791350 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet baa842552d00489998f83e68aa0d516b (table default.Table_5fc65aee40154e6296dc8b3f3076a4f3 [id=13e03112a7a641828153301cefc1774e]) successfully deleted
I20240507 08:11:55.791411 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.791519 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:55.791178 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 214) took 2902 ms. Trace:
I20240507 08:11:55.791720 5841 rpcz_store.cc:276] 0507 08:11:52.888312 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.888386 (+ 74us) service_pool.cc:225] Handling call
0507 08:11:55.791013 (+2902627us) ts_tablet_manager.cc:1120] Deleting tablet 6bf34089e5ec419f97c0da9ec42d7eec
0507 08:11:55.791023 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.791144 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2890748}
W20240507 08:11:55.792150 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6bf34089e5ec419f97c0da9ec42d7eec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bf34089e5ec419f97c0da9ec42d7eec already in progress: creating tablet
I20240507 08:11:55.792196 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 215) took 2897 ms. Trace:
I20240507 08:11:55.792295 5841 rpcz_store.cc:276] 0507 08:11:52.901271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.901320 (+ 49us) service_pool.cc:225] Handling call
0507 08:11:55.792084 (+2890764us) ts_tablet_manager.cc:1120] Deleting tablet 191adef34efa4c3999d4642a80177294
0507 08:11:55.792093 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.792170 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2847680}
W20240507 08:11:55.792477 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b0b9f85cd78748d3afce83b64b3f823b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b9f85cd78748d3afce83b64b3f823b already in progress: creating tablet
W20240507 08:11:55.792794 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 191adef34efa4c3999d4642a80177294 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 191adef34efa4c3999d4642a80177294 already in progress: creating tablet
I20240507 08:11:55.792853 5841 tablet_replica.cc:330] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.792642 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9ba24c078af9472da06793c58f147290
I20240507 08:11:55.793107 5841 raft_consensus.cc:2229] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.793378 5841 raft_consensus.cc:2258] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.793725 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba24c078af9472da06793c58f147290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:55.793751 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba24c078af9472da06793c58f147290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:55.794273 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9ba24c078af9472da06793c58f147290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ba24c078af9472da06793c58f147290 already in progress: creating tablet
I20240507 08:11:55.796577 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f2d4ce8cce7c491981416e3efa5a6d11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.797220 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2d4ce8cce7c491981416e3efa5a6d11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.797847 5841 ts_tablet_manager.cc:1875] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.800204 5839 tablet_bootstrap.cc:492] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.800360 5839 ts_tablet_manager.cc:1367] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.005s
I20240507 08:11:55.801157 5839 raft_consensus.cc:348] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.801399 5839 raft_consensus.cc:374] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.801512 5839 raft_consensus.cc:729] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.801791 5839 consensus_queue.cc:260] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.802455 5839 ts_tablet_manager.cc:1398] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:55.802732 5839 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.804528 5839 tablet_bootstrap.cc:654] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.804845 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6616660bc4a6400a9eb644590ced0e21" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:55.805652 5517 leader_election.cc:302] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:55.806160 6073 raft_consensus.cc:2726] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.808005 5585 raft_consensus.cc:3037] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:55.808110 5843 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.808259 5843 ts_tablet_manager.cc:1367] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.002s
I20240507 08:11:55.809024 5843 raft_consensus.cc:348] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.809280 5843 raft_consensus.cc:374] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.809388 5843 raft_consensus.cc:729] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.809448 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcefbe3a60f84cadb5fd3e69c6efdd02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.809715 5843 consensus_queue.cc:260] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.810060 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 (DEFAULT_TABLE table=default.Table_4d38bf47908640beb3f6d94ad4446717 [id=84ee77d848974c2d8ccd5d9d518e1ec2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.810402 5843 ts_tablet_manager.cc:1398] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:55.810772 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba24c078af9472da06793c58f147290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.810846 5843 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.811208 5696 tablet_service.cc:1467] Processing CreateTablet for tablet f2de26190bc44916809856d84ad68345 (DEFAULT_TABLE table=default.Table_f2eabb2165474aadb0528bf86cbc1718 [id=ff9529d9a26e41cb9ebf472b849d6764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.811509 5837 tablet_replica.cc:330] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:55.812564 5843 tablet_bootstrap.cc:654] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.812636 5716 raft_consensus.cc:2454] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:55.813810 5377 leader_election.cc:302] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.814402 6155 raft_consensus.cc:2781] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:55.814585 6155 raft_consensus.cc:686] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:55.814904 6155 consensus_queue.cc:237] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.816365 5837 raft_consensus.cc:2229] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.816556 5837 raft_consensus.cc:2258] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.816749 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.817109 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 0c4cd7b56d4e468eb754d9653117d023 (DEFAULT_TABLE table=default.Table_c1bde6a097f946a1bae0b57150f3aef5 [id=aac50aefe6524a84b1e4564d3f0ce394]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.818120 5837 ts_tablet_manager.cc:1875] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.818575 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c4cd7b56d4e468eb754d9653117d023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.821733 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2de26190bc44916809856d84ad68345. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.821856 5839 tablet_bootstrap.cc:492] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.822006 5839 ts_tablet_manager.cc:1367] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.002s
I20240507 08:11:55.822811 5839 raft_consensus.cc:348] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.823024 5839 raft_consensus.cc:374] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.823089 5839 raft_consensus.cc:729] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.823316 5839 consensus_queue.cc:260] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.823827 5839 ts_tablet_manager.cc:1398] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:11:55.824294 5839 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:55.825917 5839 tablet_bootstrap.cc:654] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.827291 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 767905763ba14cb8bbc52151ae9f17e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.836185 5841 ts_tablet_manager.cc:1888] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.836341 5841 log.cc:1196] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/add94bcf0c844cb78059b7240c7e28c0
I20240507 08:11:55.836679 5841 ts_tablet_manager.cc:1909] T add94bcf0c844cb78059b7240c7e28c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.837394 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 216) took 2896 ms. Trace:
I20240507 08:11:55.837469 5841 rpcz_store.cc:276] 0507 08:11:52.940555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.940618 (+ 63us) service_pool.cc:225] Handling call
0507 08:11:55.792772 (+2852154us) ts_tablet_manager.cc:1120] Deleting tablet add94bcf0c844cb78059b7240c7e28c0
0507 08:11:55.792784 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.801635 (+ 8851us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.836080 (+ 34445us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.837334 (+ 1254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2844073}
I20240507 08:11:55.837862 5841 tablet_replica.cc:330] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.837909 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet add94bcf0c844cb78059b7240c7e28c0 (table default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:55.838225 5841 raft_consensus.cc:2229] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.838532 5841 raft_consensus.cc:2258] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.839380 5841 ts_tablet_manager.cc:1875] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.847288 5841 ts_tablet_manager.cc:1888] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.847510 5841 log.cc:1196] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/80d82625304942d38b0261f95e235a41
I20240507 08:11:55.848047 5841 ts_tablet_manager.cc:1909] T 80d82625304942d38b0261f95e235a41 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.848304 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:55.849069 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 217) took 2902 ms. Trace:
I20240507 08:11:55.849267 5841 rpcz_store.cc:276] 0507 08:11:52.947043 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.947102 (+ 59us) service_pool.cc:225] Handling call
0507 08:11:55.837779 (+2890677us) ts_tablet_manager.cc:1120] Deleting tablet 80d82625304942d38b0261f95e235a41
0507 08:11:55.837789 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.843503 (+ 5714us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.847165 (+ 3662us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.849027 (+ 1862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2888833}
I20240507 08:11:55.849853 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 218) took 2901 ms. Trace:
I20240507 08:11:55.849932 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 80d82625304942d38b0261f95e235a41 (table default.Table_014d10f31cfe4b05a42d8f9c3d3921e3 [id=478717ae383042c8848a113d90cc85ed]) successfully deleted
I20240507 08:11:55.850024 5841 rpcz_store.cc:276] 0507 08:11:52.948027 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.948072 (+ 45us) service_pool.cc:225] Handling call
0507 08:11:55.849692 (+2901620us) ts_tablet_manager.cc:1120] Deleting tablet d537bed29a2f47058255d2530b5c0d23
0507 08:11:55.849702 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.849814 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2883002}
I20240507 08:11:55.850612 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 227) took 2730 ms. Trace:
W20240507 08:11:55.850641 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d537bed29a2f47058255d2530b5c0d23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d537bed29a2f47058255d2530b5c0d23 already in progress: creating tablet
I20240507 08:11:55.850800 5841 rpcz_store.cc:276] 0507 08:11:53.122287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.122355 (+ 68us) service_pool.cc:225] Handling call
0507 08:11:55.850478 (+2728123us) ts_tablet_manager.cc:1120] Deleting tablet cb7acbdb357445adb9ce69103c782a7f
0507 08:11:55.850491 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.850584 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2705506}
I20240507 08:11:55.851270 5841 tablet_replica.cc:330] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:11:55.851420 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cb7acbdb357445adb9ce69103c782a7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb7acbdb357445adb9ce69103c782a7f already in progress: creating tablet
I20240507 08:11:55.851637 5841 raft_consensus.cc:2229] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:55.852085 5841 pending_rounds.cc:62] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:11:55.852241 5841 pending_rounds.cc:68] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924730754326528 op_type: NO_OP noop_request { }
I20240507 08:11:55.853020 5841 raft_consensus.cc:2866] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:55.853655 5841 raft_consensus.cc:2258] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.856971 5841 ts_tablet_manager.cc:1875] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.861163 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 645ae26bb26f43d298afe98817d88fa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.862452 5843 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:55.862634 5843 ts_tablet_manager.cc:1367] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.052s user 0.007s sys 0.020s
I20240507 08:11:55.864790 5841 ts_tablet_manager.cc:1888] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:55.864900 5841 log.cc:1196] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d10b38120dfe4b9ab3c39cf584e4ff13
I20240507 08:11:55.865252 5841 ts_tablet_manager.cc:1909] T d10b38120dfe4b9ab3c39cf584e4ff13 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.866008 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 228) took 2716 ms. Trace:
I20240507 08:11:55.866123 5841 rpcz_store.cc:276] 0507 08:11:53.150001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.150214 (+ 213us) service_pool.cc:225] Handling call
0507 08:11:55.851209 (+2700995us) ts_tablet_manager.cc:1120] Deleting tablet d10b38120dfe4b9ab3c39cf584e4ff13
0507 08:11:55.851218 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.862327 (+ 11109us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.864694 (+ 2367us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.865968 (+ 1274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2681422}
I20240507 08:11:55.866591 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 226) took 2752 ms. Trace:
I20240507 08:11:55.866570 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d10b38120dfe4b9ab3c39cf584e4ff13 (table default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]) successfully deleted
I20240507 08:11:55.866783 5841 rpcz_store.cc:276] 0507 08:11:53.113942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.115893 (+ 1951us) service_pool.cc:225] Handling call
0507 08:11:55.866427 (+2750534us) ts_tablet_manager.cc:1120] Deleting tablet bde62374dc7f4a67a071b6b658e501c4
0507 08:11:55.866435 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.866558 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2666345}
I20240507 08:11:55.867165 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 231) took 2675 ms. Trace:
W20240507 08:11:55.867203 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bde62374dc7f4a67a071b6b658e501c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bde62374dc7f4a67a071b6b658e501c4 already in progress: creating tablet
I20240507 08:11:55.867285 5841 rpcz_store.cc:276] 0507 08:11:53.191342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.191468 (+ 126us) service_pool.cc:225] Handling call
0507 08:11:55.867019 (+2675551us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:11:55.867027 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.867128 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619585}
W20240507 08:11:55.867614 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 193ec0b463d04835b74df80229c85cde because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 193ec0b463d04835b74df80229c85cde already in progress: creating tablet
I20240507 08:11:55.867671 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 230) took 2682 ms. Trace:
I20240507 08:11:55.867759 5841 rpcz_store.cc:276] 0507 08:11:53.185632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.185857 (+ 225us) service_pool.cc:225] Handling call
0507 08:11:55.867510 (+2681653us) ts_tablet_manager.cc:1120] Deleting tablet 107d9b3f95e14044854ad0bf91a268ee
0507 08:11:55.867519 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.867643 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2609463}
I20240507 08:11:55.868101 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 242) took 2372 ms. Trace:
W20240507 08:11:55.868135 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 107d9b3f95e14044854ad0bf91a268ee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 107d9b3f95e14044854ad0bf91a268ee already in progress: creating tablet
I20240507 08:11:55.868741 5841 rpcz_store.cc:276] 0507 08:11:53.496020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.496172 (+ 152us) service_pool.cc:225] Handling call
0507 08:11:55.867981 (+2371809us) ts_tablet_manager.cc:1120] Deleting tablet 168e49ae24314a7aae2291485ddeda44
0507 08:11:55.867988 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.868062 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2371423}
I20240507 08:11:55.869118 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 244) took 2341 ms. Trace:
W20240507 08:11:55.869117 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 168e49ae24314a7aae2291485ddeda44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 168e49ae24314a7aae2291485ddeda44 already in progress: creating tablet
I20240507 08:11:55.869236 5841 rpcz_store.cc:276] 0507 08:11:53.527797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.527847 (+ 50us) service_pool.cc:225] Handling call
0507 08:11:55.868999 (+2341152us) ts_tablet_manager.cc:1120] Deleting tablet 1b19d8e28f164c588946ea5e81ec320e
0507 08:11:55.869007 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.869080 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2340260}
I20240507 08:11:55.869527 5841 tablet_replica.cc:330] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:11:55.869686 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1b19d8e28f164c588946ea5e81ec320e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1b19d8e28f164c588946ea5e81ec320e already in progress: creating tablet
I20240507 08:11:55.869761 5841 raft_consensus.cc:2229] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.869920 5841 raft_consensus.cc:2258] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:55.870469 5841 ts_tablet_manager.cc:1875] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.871176 5837 ts_tablet_manager.cc:1888] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.871366 5837 log.cc:1196] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/9ba24c078af9472da06793c58f147290
I20240507 08:11:55.871775 5837 ts_tablet_manager.cc:1909] T 9ba24c078af9472da06793c58f147290 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:55.868783 5843 raft_consensus.cc:348] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.873766 5843 raft_consensus.cc:374] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.873847 5843 raft_consensus.cc:729] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.874167 5843 consensus_queue.cc:260] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.875046 5843 ts_tablet_manager.cc:1398] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240507 08:11:55.875391 5843 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:55.877321 5843 tablet_bootstrap.cc:654] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.881461 5841 ts_tablet_manager.cc:1888] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:55.881635 5841 log.cc:1196] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/812dd1ebd42f4e929838a0535c22ad69
I20240507 08:11:55.881989 5841 ts_tablet_manager.cc:1909] T 812dd1ebd42f4e929838a0535c22ad69 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:55.882727 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 243) took 2355 ms. Trace:
I20240507 08:11:55.882834 5841 rpcz_store.cc:276] 0507 08:11:53.527660 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.532323 (+ 4663us) service_pool.cc:225] Handling call
0507 08:11:55.869464 (+2337141us) ts_tablet_manager.cc:1120] Deleting tablet 812dd1ebd42f4e929838a0535c22ad69
0507 08:11:55.869472 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.878319 (+ 8847us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.881344 (+ 3025us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:55.882670 (+ 1326us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326969}
I20240507 08:11:55.883298 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 245) took 2334 ms. Trace:
I20240507 08:11:55.883265 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 812dd1ebd42f4e929838a0535c22ad69 (table default.Table_f29e8299e1fc4d2ba2b6cd3036641274 [id=a764e2dd5a9841fe9ebc72c179b7858e]) successfully deleted
I20240507 08:11:55.883386 5841 rpcz_store.cc:276] 0507 08:11:53.548620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.552076 (+ 3456us) service_pool.cc:225] Handling call
0507 08:11:55.883122 (+2331046us) ts_tablet_manager.cc:1120] Deleting tablet 8e3ffaf1781e4b108f305346476905be
0507 08:11:55.883132 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.883248 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2330751}
I20240507 08:11:55.883734 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 246) took 2157 ms. Trace:
I20240507 08:11:55.883896 5841 rpcz_store.cc:276] 0507 08:11:53.726034 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.726091 (+ 57us) service_pool.cc:225] Handling call
0507 08:11:55.883582 (+2157491us) ts_tablet_manager.cc:1120] Deleting tablet b22334378eec4ab3885d0f9d2d812a3e
0507 08:11:55.883589 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.883679 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2069913}
W20240507 08:11:55.883977 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8e3ffaf1781e4b108f305346476905be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e3ffaf1781e4b108f305346476905be already in progress: creating tablet
W20240507 08:11:55.884198 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b22334378eec4ab3885d0f9d2d812a3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22334378eec4ab3885d0f9d2d812a3e already in progress: creating tablet
I20240507 08:11:55.884258 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 250) took 2068 ms. Trace:
I20240507 08:11:55.884330 5841 rpcz_store.cc:276] 0507 08:11:53.815464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.815511 (+ 47us) service_pool.cc:225] Handling call
0507 08:11:55.884118 (+2068607us) ts_tablet_manager.cc:1120] Deleting tablet 4867e1cca1634b4ab6e36f9ed3204e0a
0507 08:11:55.884128 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.884202 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068383}
I20240507 08:11:55.884728 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 249) took 2069 ms. Trace:
W20240507 08:11:55.884730 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4867e1cca1634b4ab6e36f9ed3204e0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4867e1cca1634b4ab6e36f9ed3204e0a already in progress: creating tablet
I20240507 08:11:55.884817 5841 rpcz_store.cc:276] 0507 08:11:53.815331 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.815774 (+ 443us) service_pool.cc:225] Handling call
0507 08:11:55.884587 (+2068813us) ts_tablet_manager.cc:1120] Deleting tablet 8ed8bac124cb4de983f0fdbaf728975f
0507 08:11:55.884595 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.884670 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068655}
I20240507 08:11:55.885118 5841 tablet_replica.cc:330] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:55.885010 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ac1bf080fe40499889f843cc9bf7e236" } new_table_name: "default.Table_3d84936f1b42408a983ad41684136fb3" modify_external_catalogs: true
I20240507 08:11:55.885313 5841 raft_consensus.cc:2229] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:55.885496 5841 raft_consensus.cc:2258] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:55.886938 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8ed8bac124cb4de983f0fdbaf728975f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed8bac124cb4de983f0fdbaf728975f already in progress: creating tablet
I20240507 08:11:55.890893 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 (DEFAULT_TABLE table=default.Table_54a33e3ca40248028587733beb4f6743 [id=9b06744f353d42a69ea7f5b5b60f3a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.890991 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_221bfcf09b8e4d5e95e2ab49ad514c10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.891314 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:55.891714 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_221bfcf09b8e4d5e95e2ab49ad514c10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.893440 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0eea053201704cd3a7492f74e4b3340e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.893978 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0eea053201704cd3a7492f74e4b3340e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.895109 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet a7df45c39d1b490db97c72a0eb6cf3ba from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:55.895216 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3ffbb2e35cc34f288580fefe1e79ebfd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:55.895821 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ffbb2e35cc34f288580fefe1e79ebfd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.899478 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01e3311a8b4553a31914922fb08d13 (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.900032 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01e3311a8b4553a31914922fb08d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.900249 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce (DEFAULT_TABLE table=default.Table_bc44d932cc29477e8751f038712ef26c [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.900615 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc9bf1c04e4015a152cb750184c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.901638 5840 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.902081 5840 ts_tablet_manager.cc:1367] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.270s user 0.030s sys 0.045s
I20240507 08:11:55.903043 5840 raft_consensus.cc:348] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.903331 5840 raft_consensus.cc:374] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.903445 5840 raft_consensus.cc:729] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.903409 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce (DEFAULT_TABLE table=default.Table_bc44d932cc29477e8751f038712ef26c [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.903892 5840 consensus_queue.cc:260] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.904368 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc9bf1c04e4015a152cb750184c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.905843 5549 tablet_service.cc:1467] Processing CreateTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 (DEFAULT_TABLE table=default.Table_3740c610e0f047f4b335d6e49868a581 [id=2a51124d3ab54fc58394e23a6aead27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.906211 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ebbcf0d5a54d63a96a9374e839af83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.907183 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2da5736a4b6944c29d94f8b89cc4e638"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:55.907440 5415 tablet_service.cc:1467] Processing CreateTablet for tablet b1fadbfd330b4e699e151f83125da281 (DEFAULT_TABLE table=default.Table_307c9baefe434bb68c8d0906a9b9788e [id=6415c5a6c29747b6b005de15fb2ab365]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:55.907733 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2da5736a4b6944c29d94f8b89cc4e638 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:55.907794 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fadbfd330b4e699e151f83125da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.908701 5840 ts_tablet_manager.cc:1398] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:55.909868 5559 tablet_service.cc:1467] Processing CreateTablet for tablet b1fadbfd330b4e699e151f83125da281 (DEFAULT_TABLE table=default.Table_307c9baefe434bb68c8d0906a9b9788e [id=6415c5a6c29747b6b005de15fb2ab365]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.910218 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fadbfd330b4e699e151f83125da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.910346 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01e3311a8b4553a31914922fb08d13 (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.910705 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01e3311a8b4553a31914922fb08d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.913178 5840 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.913928 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 (DEFAULT_TABLE table=default.Table_54a33e3ca40248028587733beb4f6743 [id=9b06744f353d42a69ea7f5b5b60f3a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.914462 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.914477 5413 tablet_service.cc:1467] Processing CreateTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 (DEFAULT_TABLE table=default.Table_3740c610e0f047f4b335d6e49868a581 [id=2a51124d3ab54fc58394e23a6aead27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.914548 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.914844 5840 tablet_bootstrap.cc:654] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.914846 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ebbcf0d5a54d63a96a9374e839af83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.915278 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.915649 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a30bfc61b4ff2be9f2418ca59b030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.918673 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a30bfc61b4ff2be9f2418ca59b030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.921149 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 3f01e3311a8b4553a31914922fb08d13 (DEFAULT_TABLE table=default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.929073 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.929497 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 055a30bfc61b4ff2be9f2418ca59b030. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.934332 5698 tablet_service.cc:1467] Processing CreateTablet for tablet b1fadbfd330b4e699e151f83125da281 (DEFAULT_TABLE table=default.Table_307c9baefe434bb68c8d0906a9b9788e [id=6415c5a6c29747b6b005de15fb2ab365]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.934795 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1fadbfd330b4e699e151f83125da281. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.935417 5840 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.935577 5840 ts_tablet_manager.cc:1367] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.004s
I20240507 08:11:55.936457 5840 raft_consensus.cc:348] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.936681 5840 raft_consensus.cc:374] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.936808 5840 raft_consensus.cc:729] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.941630 5840 consensus_queue.cc:260] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.942806 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce (DEFAULT_TABLE table=default.Table_bc44d932cc29477e8751f038712ef26c [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.945091 5840 ts_tablet_manager.cc:1398] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20240507 08:11:55.946187 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53bc9bf1c04e4015a152cb750184c9ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.955447 5840 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:55.956283 6087 raft_consensus.cc:482] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.956418 6087 raft_consensus.cc:504] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.957059 6087 leader_election.cc:288] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:55.958173 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca8fc2c7f3849c380fea56862ffc26e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:55.958738 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 44ec0526e1914221bc22b3e75ea01e0d
W20240507 08:11:55.958881 5654 leader_election.cc:341] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.960985 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_94b53721a4784e1b94415eb370b8dd9c" } new_table_name: "default.Table_086d5c28a17243c9991e0add9910274a" modify_external_catalogs: true
I20240507 08:11:55.963121 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:55.963958 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 44ec0526e1914221bc22b3e75ea01e0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ec0526e1914221bc22b3e75ea01e0d already in progress: creating tablet
I20240507 08:11:55.965145 5840 tablet_bootstrap.cc:654] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:55.966172 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 9ba24c078af9472da06793c58f147290 (table default.Table_70320eb4c56d4c38b5f30235746e7cb2 [id=8ea36271c6814ba58435556d3ff59f2a]) successfully deleted
W20240507 08:11:55.968799 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b0b9f85cd78748d3afce83b64b3f823b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b9f85cd78748d3afce83b64b3f823b already in progress: creating tablet
W20240507 08:11:55.969146 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 753019050fcf4dc3b69722c3eb475037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753019050fcf4dc3b69722c3eb475037 already in progress: creating tablet
I20240507 08:11:55.971419 5841 ts_tablet_manager.cc:1875] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:55.979432 5699 tablet_service.cc:1467] Processing CreateTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 (DEFAULT_TABLE table=default.Table_3740c610e0f047f4b335d6e49868a581 [id=2a51124d3ab54fc58394e23a6aead27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.979583 5839 tablet_bootstrap.cc:492] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:55.979738 5839 ts_tablet_manager.cc:1367] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.156s user 0.014s sys 0.022s
I20240507 08:11:55.980156 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8ebbcf0d5a54d63a96a9374e839af83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.980446 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca8fc2c7f3849c380fea56862ffc26e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:55.980970 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:55.981006 5656 leader_election.cc:341] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:55.981174 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f01e3311a8b4553a31914922fb08d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.981146 5656 leader_election.cc:302] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:55.981555 6087 raft_consensus.cc:2726] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:55.987952 6155 raft_consensus.cc:482] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:55.988134 6155 raft_consensus.cc:504] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.989183 5840 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:55.989399 5840 ts_tablet_manager.cc:1367] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.005s
I20240507 08:11:55.989809 5839 raft_consensus.cc:348] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.990113 5839 raft_consensus.cc:374] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.990223 5839 raft_consensus.cc:729] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.990509 5840 raft_consensus.cc:348] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.990816 5840 raft_consensus.cc:374] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:55.990949 5840 raft_consensus.cc:729] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:55.991041 5839 consensus_queue.cc:260] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:55.991358 5840 consensus_queue.cc:260] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:55.992635 5839 ts_tablet_manager.cc:1398] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240507 08:11:55.995488 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 8a015bf84a454d49b6497c16ed15c84a (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.996116 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a015bf84a454d49b6497c16ed15c84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:55.997161 5840 ts_tablet_manager.cc:1398] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240507 08:11:55.998112 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 8a015bf84a454d49b6497c16ed15c84a (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:55.998536 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a015bf84a454d49b6497c16ed15c84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.000070 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 8a015bf84a454d49b6497c16ed15c84a (DEFAULT_TABLE table=default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.000465 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a015bf84a454d49b6497c16ed15c84a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.000766 5839 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.002166 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.004027 5839 tablet_bootstrap.cc:654] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.004081 5716 raft_consensus.cc:1261] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:56.004945 5841 ts_tablet_manager.cc:1888] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.005111 5841 log.cc:1196] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/697611671ef64c53b0be4f5341b8f270
I20240507 08:11:56.005023 5995 consensus_queue.cc:1035] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:56.008667 5840 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.010039 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f29568ebe4774662a8b49b7829cf30e7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:56.010404 5840 tablet_bootstrap.cc:654] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.009959 6155 leader_election.cc:288] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:56.011183 5843 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:11:56.010172 5377 leader_election.cc:341] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: f29568ebe4774662a8b49b7829cf30e7
I20240507 08:11:56.012034 5843 ts_tablet_manager.cc:1367] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.137s user 0.019s sys 0.035s
W20240507 08:11:56.012075 5378 leader_election.cc:341] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.012328 5378 leader_election.cc:302] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.012857 5843 raft_consensus.cc:348] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.013073 5843 raft_consensus.cc:374] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.013007 6155 raft_consensus.cc:2726] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.013188 5843 raft_consensus.cc:729] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.013464 5843 consensus_queue.cc:260] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.014245 5843 ts_tablet_manager.cc:1398] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:56.005556 5841 ts_tablet_manager.cc:1909] T 697611671ef64c53b0be4f5341b8f270 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.015483 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 248) took 2210 ms. Trace:
I20240507 08:11:56.015641 5841 rpcz_store.cc:276] 0507 08:11:53.805277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.805338 (+ 61us) service_pool.cc:225] Handling call
0507 08:11:55.885039 (+2079701us) ts_tablet_manager.cc:1120] Deleting tablet 697611671ef64c53b0be4f5341b8f270
0507 08:11:55.885047 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:55.996831 (+111784us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.004278 (+ 7447us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.015439 (+ 11161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2062268}
I20240507 08:11:56.017040 6179 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:56.017490 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 697611671ef64c53b0be4f5341b8f270 (table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]) successfully deleted
I20240507 08:11:56.024868 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 252) took 2201 ms. Trace:
I20240507 08:11:56.024994 5841 rpcz_store.cc:276] 0507 08:11:53.823292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.829441 (+ 6149us) service_pool.cc:225] Handling call
0507 08:11:56.024708 (+2195267us) ts_tablet_manager.cc:1120] Deleting tablet b4504b1ed3d449619926fe80469d9790
0507 08:11:56.024719 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.024831 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2195034}
I20240507 08:11:56.025718 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 261) took 2117 ms. Trace:
I20240507 08:11:56.025827 5841 rpcz_store.cc:276] 0507 08:11:53.908669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.908764 (+ 95us) service_pool.cc:225] Handling call
0507 08:11:56.025549 (+2116785us) ts_tablet_manager.cc:1120] Deleting tablet 88a2cfb5f046434c8340234a447a7e65
0507 08:11:56.025559 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.025686 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2108288}
I20240507 08:11:56.026091 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet fde460fdcf254af6a3cdaaac22ac2d93
W20240507 08:11:56.026154 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43644b4405f44bfd9a1afcaedbd68db5 consensus state has no leader
I20240507 08:11:56.026449 5843 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.026890 5264 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 970619b17f7c41fc95eeef72112adb5e
I20240507 08:11:56.028048 5843 tablet_bootstrap.cc:654] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.028816 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 260) took 2122 ms. Trace:
I20240507 08:11:56.028928 5841 rpcz_store.cc:276] 0507 08:11:53.906261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.906348 (+ 87us) service_pool.cc:225] Handling call
0507 08:11:56.028685 (+2122337us) ts_tablet_manager.cc:1120] Deleting tablet 5e53409fe13e4a9585dbba648a25cc07
0507 08:11:56.028694 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.028785 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2111163}
I20240507 08:11:56.029032 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:56.029670 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b4504b1ed3d449619926fe80469d9790 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4504b1ed3d449619926fe80469d9790 already in progress: creating tablet
I20240507 08:11:56.029748 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.029729 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet fde460fdcf254af6a3cdaaac22ac2d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde460fdcf254af6a3cdaaac22ac2d93 already in progress: creating tablet
I20240507 08:11:56.029927 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 262) took 2117 ms. Trace:
W20240507 08:11:56.029932 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 88a2cfb5f046434c8340234a447a7e65 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88a2cfb5f046434c8340234a447a7e65 already in progress: creating tablet
I20240507 08:11:56.030021 5841 rpcz_store.cc:276] 0507 08:11:53.912175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.912228 (+ 53us) service_pool.cc:225] Handling call
0507 08:11:56.029350 (+2117122us) ts_tablet_manager.cc:1120] Deleting tablet 67c8624285d946e597227430a5714e2f
0507 08:11:56.029358 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.029897 (+ 539us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2110002}
W20240507 08:11:56.030491 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 67c8624285d946e597227430a5714e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67c8624285d946e597227430a5714e2f already in progress: creating tablet
I20240507 08:11:56.030577 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 273) took 2037 ms. Trace:
I20240507 08:11:56.030706 5841 rpcz_store.cc:276] 0507 08:11:53.993148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:53.993211 (+ 63us) service_pool.cc:225] Handling call
0507 08:11:56.030420 (+2037209us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:11:56.030431 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.030541 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2027889}
W20240507 08:11:56.031076 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1c7c39ec9fec4331afabb39d1483ef57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c7c39ec9fec4331afabb39d1483ef57 already in progress: creating tablet
I20240507 08:11:56.031244 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 277) took 1933 ms. Trace:
I20240507 08:11:56.031390 5841 rpcz_store.cc:276] 0507 08:11:54.097675 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.097858 (+ 183us) service_pool.cc:225] Handling call
0507 08:11:56.031094 (+1933236us) ts_tablet_manager.cc:1120] Deleting tablet 907b5cb0e7a54903abe0b0d0e8d617dc
0507 08:11:56.031106 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.031207 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1932996}
W20240507 08:11:56.031806 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 907b5cb0e7a54903abe0b0d0e8d617dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 907b5cb0e7a54903abe0b0d0e8d617dc already in progress: creating tablet
I20240507 08:11:56.033089 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 278) took 1822 ms. Trace:
I20240507 08:11:56.033243 5841 rpcz_store.cc:276] 0507 08:11:54.210125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.210461 (+ 336us) service_pool.cc:225] Handling call
0507 08:11:56.032686 (+1822225us) ts_tablet_manager.cc:1120] Deleting tablet 98aa4ec7b93943e5bf8f9758b3bb71e0
0507 08:11:56.032697 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.033038 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1815327}
W20240507 08:11:56.033758 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 already in progress: creating tablet
I20240507 08:11:56.033840 5841 tablet_replica.cc:330] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.034116 5841 raft_consensus.cc:2229] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.034307 5841 raft_consensus.cc:2258] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.035013 5841 ts_tablet_manager.cc:1875] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.044621 5840 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.044780 5840 ts_tablet_manager.cc:1367] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.003s
I20240507 08:11:56.045174 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_641a95550177400e87be665acc80c76c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.045547 5840 raft_consensus.cc:348] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:56.045765 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_641a95550177400e87be665acc80c76c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.045784 5840 raft_consensus.cc:374] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.045877 5840 raft_consensus.cc:729] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.046604 5843 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.046759 5843 ts_tablet_manager.cc:1367] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240507 08:11:56.047039 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a99b17a89c46447ea9e6544b2f7eb4d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.047487 5843 raft_consensus.cc:348] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:56.047607 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.047691 5843 raft_consensus.cc:374] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.047755 5843 raft_consensus.cc:729] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.048000 5843 consensus_queue.cc:260] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.048650 5840 consensus_queue.cc:260] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.049710 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:56.049929 5843 ts_tablet_manager.cc:1398] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:56.050150 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c6cb7920df47aa97be6555c587798a with delete_type TABLET_DATA_DELETED (replaced by tablet f5e48a2b4a0e4088ad009a7f8fe77f3f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.050228 5843 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.050431 5838 tablet_replica.cc:330] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.050601 5838 raft_consensus.cc:2229] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.050902 5838 raft_consensus.cc:2258] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.051039 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a94e166f91d741cd9c5e85ea5efb8e3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.051460 5840 ts_tablet_manager.cc:1398] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
W20240507 08:11:56.051483 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a94e166f91d741cd9c5e85ea5efb8e3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.052392 5843 tablet_bootstrap.cc:654] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.054142 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_18ef77bde60b44179d9367054bffc7af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.054631 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18ef77bde60b44179d9367054bffc7af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.055265 5838 ts_tablet_manager.cc:1875] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.056206 5840 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.056205 5841 ts_tablet_manager.cc:1888] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.056344 5841 log.cc:1196] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c4899c088531480ea15364af94595ee5
I20240507 08:11:56.056746 5841 ts_tablet_manager.cc:1909] T c4899c088531480ea15364af94595ee5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.057562 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 279) took 1687 ms. Trace:
I20240507 08:11:56.057829 5840 tablet_bootstrap.cc:654] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.057698 5841 rpcz_store.cc:276] 0507 08:11:54.370013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.370375 (+ 362us) service_pool.cc:225] Handling call
0507 08:11:56.033763 (+1663388us) ts_tablet_manager.cc:1120] Deleting tablet c4899c088531480ea15364af94595ee5
0507 08:11:56.033775 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.044355 (+ 10580us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.056091 (+ 11736us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.057510 (+ 1419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1663086}
I20240507 08:11:56.058274 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 280) took 1688 ms. Trace:
I20240507 08:11:56.058378 5841 rpcz_store.cc:276] 0507 08:11:54.370182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.370867 (+ 685us) service_pool.cc:225] Handling call
0507 08:11:56.058107 (+1687240us) ts_tablet_manager.cc:1120] Deleting tablet f49dd4772f8e46478638b29c009c83ea
0507 08:11:56.058117 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.058223 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687040}
I20240507 08:11:56.058985 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 281) took 1688 ms. Trace:
I20240507 08:11:56.059087 5841 rpcz_store.cc:276] 0507 08:11:54.370300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.371099 (+ 799us) service_pool.cc:225] Handling call
0507 08:11:56.058738 (+1687639us) ts_tablet_manager.cc:1120] Deleting tablet c43f3de489904f2ba2a637d408619bfc
0507 08:11:56.058835 (+ 97us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.058931 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1687487}
I20240507 08:11:56.059537 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 288) took 1548 ms. Trace:
I20240507 08:11:56.059621 5841 rpcz_store.cc:276] 0507 08:11:54.511269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.513630 (+ 2361us) service_pool.cc:225] Handling call
0507 08:11:56.059405 (+1545775us) ts_tablet_manager.cc:1120] Deleting tablet 34bebec088a34ebbb06d2e707d54da4c
0507 08:11:56.059413 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.059487 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545601}
I20240507 08:11:56.060314 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 290) took 1538 ms. Trace:
I20240507 08:11:56.060401 5841 rpcz_store.cc:276] 0507 08:11:54.521333 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.522119 (+ 786us) service_pool.cc:225] Handling call
0507 08:11:56.059951 (+1537832us) ts_tablet_manager.cc:1120] Deleting tablet 7eaf0e2936324a9fbb2426a8026886fc
0507 08:11:56.059960 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.060274 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537571}
I20240507 08:11:56.060835 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 291) took 1539 ms. Trace:
I20240507 08:11:56.060909 5841 rpcz_store.cc:276] 0507 08:11:54.521467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.522532 (+ 1065us) service_pool.cc:225] Handling call
0507 08:11:56.060687 (+1538155us) ts_tablet_manager.cc:1120] Deleting tablet d45d3fbbe3764261a3a24523a77c6443
0507 08:11:56.060695 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.060781 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537970}
I20240507 08:11:56.061290 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 289) took 1540 ms. Trace:
I20240507 08:11:56.061370 5841 rpcz_store.cc:276] 0507 08:11:54.521177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.526130 (+ 4953us) service_pool.cc:225] Handling call
0507 08:11:56.061174 (+1535044us) ts_tablet_manager.cc:1120] Deleting tablet 7ba2e8d1e01047fd9cc1c192e9628750
0507 08:11:56.061181 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.061248 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534841}
I20240507 08:11:56.061858 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 292) took 1540 ms. Trace:
I20240507 08:11:56.061947 5841 rpcz_store.cc:276] 0507 08:11:54.521565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.526775 (+ 5210us) service_pool.cc:225] Handling call
0507 08:11:56.061671 (+1534896us) ts_tablet_manager.cc:1120] Deleting tablet 87377661d9374cef8ab1a97042b7f1b9
0507 08:11:56.061681 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.061804 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534665}
I20240507 08:11:56.062336 5841 tablet_replica.cc:330] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.062522 5841 raft_consensus.cc:2229] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.062693 5841 raft_consensus.cc:2258] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.063217 5841 ts_tablet_manager.cc:1875] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.066920 5838 ts_tablet_manager.cc:1888] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.067090 5838 log.cc:1196] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/91c6cb7920df47aa97be6555c587798a
I20240507 08:11:56.067298 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c4899c088531480ea15364af94595ee5 (table default.Table_8ad438587c5c48c78d151533b16a55e3 [id=307b8b3612c1470d916ad0f37c986171]) successfully deleted
I20240507 08:11:56.067476 5838 ts_tablet_manager.cc:1909] T 91c6cb7920df47aa97be6555c587798a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.067540 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.067672 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f49dd4772f8e46478638b29c009c83ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f49dd4772f8e46478638b29c009c83ea already in progress: creating tablet
I20240507 08:11:56.067790 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.067937 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c43f3de489904f2ba2a637d408619bfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c43f3de489904f2ba2a637d408619bfc already in progress: creating tablet
W20240507 08:11:56.068145 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 34bebec088a34ebbb06d2e707d54da4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34bebec088a34ebbb06d2e707d54da4c already in progress: creating tablet
W20240507 08:11:56.068406 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d45d3fbbe3764261a3a24523a77c6443 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d45d3fbbe3764261a3a24523a77c6443 already in progress: creating tablet
I20240507 08:11:56.068526 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 91c6cb7920df47aa97be6555c587798a (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
W20240507 08:11:56.068651 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7ba2e8d1e01047fd9cc1c192e9628750 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ba2e8d1e01047fd9cc1c192e9628750 already in progress: creating tablet
W20240507 08:11:56.068854 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 87377661d9374cef8ab1a97042b7f1b9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87377661d9374cef8ab1a97042b7f1b9 already in progress: creating tablet
I20240507 08:11:56.070601 5841 ts_tablet_manager.cc:1888] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.070713 5841 log.cc:1196] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/0a6a35991c18414b9bc714e9a0ba14d8
I20240507 08:11:56.071050 5841 ts_tablet_manager.cc:1909] T 0a6a35991c18414b9bc714e9a0ba14d8 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.071743 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 293) took 1523 ms. Trace:
I20240507 08:11:56.071866 5841 rpcz_store.cc:276] 0507 08:11:54.547826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.548546 (+ 720us) service_pool.cc:225] Handling call
0507 08:11:56.062247 (+1513701us) ts_tablet_manager.cc:1120] Deleting tablet 0a6a35991c18414b9bc714e9a0ba14d8
0507 08:11:56.062254 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.066613 (+ 4359us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.070491 (+ 3878us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.071708 (+ 1217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513485}
I20240507 08:11:56.072253 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 0a6a35991c18414b9bc714e9a0ba14d8 (table default.Table_fd604f4f74ed4e14b1bd8df5fb3731a1 [id=5c0760df27cc4b34963feb5cecd6372b]) successfully deleted
I20240507 08:11:56.072393 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 302) took 1473 ms. Trace:
I20240507 08:11:56.072504 5841 rpcz_store.cc:276] 0507 08:11:54.598802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.598855 (+ 53us) service_pool.cc:225] Handling call
0507 08:11:56.072242 (+1473387us) ts_tablet_manager.cc:1120] Deleting tablet 1cd02a49359743c4b4e540ab0338d8ed
0507 08:11:56.072251 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.072363 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473166}
W20240507 08:11:56.072873 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1cd02a49359743c4b4e540ab0338d8ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cd02a49359743c4b4e540ab0338d8ed already in progress: creating tablet
I20240507 08:11:56.072968 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 301) took 1478 ms. Trace:
I20240507 08:11:56.073076 5841 rpcz_store.cc:276] 0507 08:11:54.594060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.599158 (+ 5098us) service_pool.cc:225] Handling call
0507 08:11:56.072797 (+1473639us) ts_tablet_manager.cc:1120] Deleting tablet a56cb45381604ac9a69e6f3edaa4eefc
0507 08:11:56.072805 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.072942 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1473357}
W20240507 08:11:56.073480 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a56cb45381604ac9a69e6f3edaa4eefc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a56cb45381604ac9a69e6f3edaa4eefc already in progress: creating tablet
I20240507 08:11:56.073509 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 304) took 1471 ms. Trace:
I20240507 08:11:56.073652 5841 rpcz_store.cc:276] 0507 08:11:54.601758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.601803 (+ 45us) service_pool.cc:225] Handling call
0507 08:11:56.073396 (+1471593us) ts_tablet_manager.cc:1120] Deleting tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f
0507 08:11:56.073404 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.073480 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1471403}
I20240507 08:11:56.074060 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 309) took 1172 ms. Trace:
W20240507 08:11:56.074043 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f already in progress: creating tablet
I20240507 08:11:56.074168 5841 rpcz_store.cc:276] 0507 08:11:54.901205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.901273 (+ 68us) service_pool.cc:225] Handling call
0507 08:11:56.073935 (+1172662us) ts_tablet_manager.cc:1120] Deleting tablet c48da19c9b9e4207b267137d240adc5b
0507 08:11:56.073943 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.074028 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171001}
I20240507 08:11:56.074540 5841 tablet_replica.cc:330] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.074739 5841 raft_consensus.cc:2229] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.074877 5841 raft_consensus.cc:2258] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:56.075013 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c48da19c9b9e4207b267137d240adc5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c48da19c9b9e4207b267137d240adc5b already in progress: creating tablet
I20240507 08:11:56.075405 5841 ts_tablet_manager.cc:1875] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.082015 5841 ts_tablet_manager.cc:1888] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.082161 5841 log.cc:1196] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/91c6cb7920df47aa97be6555c587798a
I20240507 08:11:56.082711 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.086308 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.086625 5837 tablet_replica.cc:330] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:56.086818 5837 raft_consensus.cc:2229] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:56.086968 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 44ec0526e1914221bc22b3e75ea01e0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ec0526e1914221bc22b3e75ea01e0d already in progress: creating tablet
I20240507 08:11:56.087051 5837 raft_consensus.cc:2258] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:56.087548 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet fde460fdcf254af6a3cdaaac22ac2d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde460fdcf254af6a3cdaaac22ac2d93 already in progress: creating tablet
I20240507 08:11:56.087927 5837 ts_tablet_manager.cc:1875] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.089814 5841 ts_tablet_manager.cc:1909] T 91c6cb7920df47aa97be6555c587798a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.090929 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 310) took 1155 ms. Trace:
I20240507 08:11:56.091022 5841 rpcz_store.cc:276] 0507 08:11:54.934939 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.935623 (+ 684us) service_pool.cc:225] Handling call
0507 08:11:56.074483 (+1138860us) ts_tablet_manager.cc:1120] Deleting tablet 91c6cb7920df47aa97be6555c587798a
0507 08:11:56.074490 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.078465 (+ 3975us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.081918 (+ 3453us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:56.090872 (+ 8954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1138646}
I20240507 08:11:56.091516 5585 raft_consensus.cc:3037] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.091568 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 308) took 1191 ms. Trace:
I20240507 08:11:56.091653 5841 rpcz_store.cc:276] 0507 08:11:54.899921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.910230 (+ 10309us) service_pool.cc:225] Handling call
0507 08:11:56.091401 (+1181171us) ts_tablet_manager.cc:1120] Deleting tablet b6e68364238c4b66b8c0de0f4a2ea855
0507 08:11:56.091410 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.091516 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1150183}
W20240507 08:11:56.091958 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b6e68364238c4b66b8c0de0f4a2ea855 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6e68364238c4b66b8c0de0f4a2ea855 already in progress: creating tablet
I20240507 08:11:56.092276 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 311) took 1092 ms. Trace:
I20240507 08:11:56.092368 5841 rpcz_store.cc:276] 0507 08:11:54.999674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:54.999727 (+ 53us) service_pool.cc:225] Handling call
0507 08:11:56.092104 (+1092377us) ts_tablet_manager.cc:1120] Deleting tablet dfa14901cdf14b72bea365b04ad521bd
0507 08:11:56.092114 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:56.092219 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092161}
W20240507 08:11:56.093501 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 31251576beee4a8a81a453b721c9c058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31251576beee4a8a81a453b721c9c058 already in progress: creating tablet
W20240507 08:11:56.094058 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4de7053cfaea4a54a402c9d826613c05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4de7053cfaea4a54a402c9d826613c05 already in progress: creating tablet
W20240507 08:11:56.094975 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3b5e3d23bbb6439a87b303dff1b44a1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b5e3d23bbb6439a87b303dff1b44a1d already in progress: creating tablet
I20240507 08:11:56.094990 5585 raft_consensus.cc:1261] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240507 08:11:56.095563 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f5e48a2b4a0e4088ad009a7f8fe77f3f already in progress: creating tablet
W20240507 08:11:56.096251 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5471c74c33cd42e8aaf47f71f61adff7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5471c74c33cd42e8aaf47f71f61adff7 already in progress: creating tablet
I20240507 08:11:56.096956 6155 consensus_queue.cc:1035] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:11:56.097051 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7b4b84ab3f274185a2b838ebb969ab52 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b4b84ab3f274185a2b838ebb969ab52 already in progress: creating tablet
W20240507 08:11:56.097735 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5ca8fc2c7f3849c380fea56862ffc26e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ca8fc2c7f3849c380fea56862ffc26e already in progress: creating tablet
W20240507 08:11:56.098351 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd already in progress: creating tablet
W20240507 08:11:56.099094 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3f01e3311a8b4553a31914922fb08d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f01e3311a8b4553a31914922fb08d13 already in progress: creating tablet
W20240507 08:11:56.099807 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9e447562e82347a584e39586500c27c2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e447562e82347a584e39586500c27c2 already in progress: creating tablet
W20240507 08:11:56.100471 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 070fdbf0b00b4582a184729c2f2a4b69 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 070fdbf0b00b4582a184729c2f2a4b69 already in progress: creating tablet
I20240507 08:11:56.100561 5940 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
W20240507 08:11:56.101135 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e1f9a56a95074e67bb33ceee1af48a18 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1f9a56a95074e67bb33ceee1af48a18 already in progress: creating tablet
I20240507 08:11:56.101794 5940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
W20240507 08:11:56.102106 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 already in progress: creating tablet
W20240507 08:11:56.102866 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 973a199a533b4bbea3c2615acdbf95ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 973a199a533b4bbea3c2615acdbf95ca already in progress: creating tablet
I20240507 08:11:56.103292 5841 tablet_replica.cc:330] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.103556 5841 raft_consensus.cc:2229] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.103830 5841 raft_consensus.cc:2258] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.104058 5258 catalog_manager.cc:5484] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:56.104524 5841 ts_tablet_manager.cc:1875] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.107249 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a7df45c39d1b490db97c72a0eb6cf3ba
I20240507 08:11:56.107772 5256 catalog_manager.cc:6554] ReplaceTablet: tablet a7df45c39d1b490db97c72a0eb6cf3ba deleted and replaced by tablet 43225934cef04109b3dbc057b230f032
I20240507 08:11:56.108006 5837 ts_tablet_manager.cc:1888] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.108136 5837 log.cc:1196] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5ca8fc2c7f3849c380fea56862ffc26e
I20240507 08:11:56.108230 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.108479 5837 ts_tablet_manager.cc:1909] T 5ca8fc2c7f3849c380fea56862ffc26e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:56.109869 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:56.111138 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.111938 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.112190 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet d10b38120dfe4b9ab3c39cf584e4ff13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.112402 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet e0f3ee2b04de4cdf8a621b39f1476577 with delete_type TABLET_DATA_DELETED (replaced by tablet 270a1fd15925416483c211021549315c) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.112777 5841 ts_tablet_manager.cc:1888] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.112948 5841 log.cc:1196] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/ebc6cc72bb7e4721a8ab559102273db5
W20240507 08:11:56.113037 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a7df45c39d1b490db97c72a0eb6cf3ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7df45c39d1b490db97c72a0eb6cf3ba already in progress: creating tablet
I20240507 08:11:56.113212 5838 tablet_replica.cc:330] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.113428 5838 raft_consensus.cc:2229] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.113494 5841 ts_tablet_manager.cc:1909] T ebc6cc72bb7e4721a8ab559102273db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.113626 5838 raft_consensus.cc:2258] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.114601 5838 ts_tablet_manager.cc:1875] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.114504 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_28dc4a4ae86d46e79217cc266d671289"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.114852 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet ebc6cc72bb7e4721a8ab559102273db5 (table default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]) successfully deleted
W20240507 08:11:56.115051 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28dc4a4ae86d46e79217cc266d671289 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.115343 5841 tablet_replica.cc:330] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.115598 5841 raft_consensus.cc:2229] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.115828 5841 raft_consensus.cc:2258] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.116556 5841 ts_tablet_manager.cc:1875] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.116550 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e49d8e80075e42cead2c50333bf2c4d3" } new_table_name: "default.Table_1c03c02bf7c44286b5e1273ff2b738c0" modify_external_catalogs: true
I20240507 08:11:56.123010 5843 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.123175 5843 ts_tablet_manager.cc:1367] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.073s user 0.004s sys 0.004s
I20240507 08:11:56.123396 5841 ts_tablet_manager.cc:1888] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.123618 5841 log.cc:1196] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5cab176c4bd34f38b14ac096f79b912e
I20240507 08:11:56.124087 5841 ts_tablet_manager.cc:1909] T 5cab176c4bd34f38b14ac096f79b912e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.124117 5843 raft_consensus.cc:348] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.125317 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5ca8fc2c7f3849c380fea56862ffc26e (table default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]) successfully deleted
I20240507 08:11:56.126374 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5cab176c4bd34f38b14ac096f79b912e (table default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]) successfully deleted
I20240507 08:11:56.127269 6186 tablet.cc:1722] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:56.127825 6179 tablet.cc:1722] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:56.128265 5841 tablet_replica.cc:330] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.128473 5841 raft_consensus.cc:2229] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.128685 5841 raft_consensus.cc:2258] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.129278 5841 ts_tablet_manager.cc:1875] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.137604 5843 raft_consensus.cc:374] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.137754 5843 raft_consensus.cc:729] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.137991 5843 consensus_queue.cc:260] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.138398 5582 raft_consensus.cc:3037] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.138593 5843 ts_tablet_manager.cc:1398] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240507 08:11:56.138931 5843 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.141085 5843 tablet_bootstrap.cc:654] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.142539 5841 ts_tablet_manager.cc:1888] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.142664 5841 log.cc:1196] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/97a43c27b30a4c3cbfd80dce5f28d7c7
I20240507 08:11:56.143030 5841 ts_tablet_manager.cc:1909] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.143352 6087 raft_consensus.cc:482] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.143507 6087 raft_consensus.cc:504] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.144151 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
I20240507 08:11:56.144243 6087 leader_election.cc:288] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:56.144908 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.146123 5654 leader_election.cc:341] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.146631 6177 mvcc.cc:205] Tried to move back new op lower bound from 7024924737544478720 to 7024924734459953152. Current Snapshot: MvccSnapshot[applied={T|T < 7024924737544478720}]
I20240507 08:11:56.147019 5838 ts_tablet_manager.cc:1888] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.147167 5838 log.cc:1196] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d10b38120dfe4b9ab3c39cf584e4ff13
I20240507 08:11:56.147379 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:56.147595 5838 ts_tablet_manager.cc:1909] T d10b38120dfe4b9ab3c39cf584e4ff13 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.148767 6186 tablet.cc:1717] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Skipping requested alter to schema version 3, tablet already version 3
I20240507 08:11:56.149567 6179 tablet.cc:1717] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Skipping requested alter to schema version 3, tablet already version 3
W20240507 08:11:56.151101 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 753019050fcf4dc3b69722c3eb475037 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 753019050fcf4dc3b69722c3eb475037 already in progress: creating tablet
W20240507 08:11:56.151178 5656 leader_election.cc:341] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.151301 5656 leader_election.cc:302] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.151525 6087 raft_consensus.cc:2726] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:56.151786 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b0b9f85cd78748d3afce83b64b3f823b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0b9f85cd78748d3afce83b64b3f823b already in progress: creating tablet
I20240507 08:11:56.152127 5841 tablet_replica.cc:330] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.152381 5841 raft_consensus.cc:2229] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.152572 5841 raft_consensus.cc:2258] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.153107 6179 tablet.cc:1722] T 6616660bc4a6400a9eb644590ced0e21 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:11:56.153357 5841 ts_tablet_manager.cc:1875] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.156033 6187 tablet.cc:1722] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:56.160928 5838 tablet_replica.cc:330] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.161487 5838 raft_consensus.cc:2229] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.161715 5838 raft_consensus.cc:2258] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.164513 5838 ts_tablet_manager.cc:1875] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.166149 5841 ts_tablet_manager.cc:1888] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.166373 5841 log.cc:1196] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/9ba24c078af9472da06793c58f147290
I20240507 08:11:56.166920 5841 ts_tablet_manager.cc:1909] T 9ba24c078af9472da06793c58f147290 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.167968 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 9ba24c078af9472da06793c58f147290 (table default.Table_70320eb4c56d4c38b5f30235746e7cb2 [id=8ea36271c6814ba58435556d3ff59f2a]) successfully deleted
W20240507 08:11:56.168690 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 44ec0526e1914221bc22b3e75ea01e0d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ec0526e1914221bc22b3e75ea01e0d already in progress: creating tablet
W20240507 08:11:56.169293 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet fde460fdcf254af6a3cdaaac22ac2d93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fde460fdcf254af6a3cdaaac22ac2d93 already in progress: creating tablet
I20240507 08:11:56.169662 5841 tablet_replica.cc:330] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.169941 5841 raft_consensus.cc:2229] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.170186 5841 raft_consensus.cc:2258] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.170864 5841 ts_tablet_manager.cc:1875] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.172827 5843 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.173009 5843 ts_tablet_manager.cc:1367] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
W20240507 08:11:56.173363 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 56835.1us
I20240507 08:11:56.173770 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d10b38120dfe4b9ab3c39cf584e4ff13 (table default.Table_6348e5422e1d4fedaac4c6084f846b96 [id=0b0951afcd524c78962cc6b058af8970]) successfully deleted
I20240507 08:11:56.176043 5839 tablet_bootstrap.cc:492] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.176221 5839 ts_tablet_manager.cc:1367] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.175s user 0.009s sys 0.021s
I20240507 08:11:56.178025 5841 ts_tablet_manager.cc:1888] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.178200 5841 log.cc:1196] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:56.178700 5841 ts_tablet_manager.cc:1909] T e49dfd218a24464fbac39a3f4b477462 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.178787 5839 raft_consensus.cc:348] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.179061 5839 raft_consensus.cc:374] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.180119 5839 raft_consensus.cc:729] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.180323 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e49dfd218a24464fbac39a3f4b477462 (table default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]) successfully deleted
I20240507 08:11:56.180440 5839 consensus_queue.cc:260] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.180831 5841 tablet_replica.cc:330] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.181074 5841 raft_consensus.cc:2229] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.181339 5841 raft_consensus.cc:2258] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.181393 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3bb889d1fe284f479791f644c016b27a" } new_table_name: "default.Table_457ae0dd8b974005872846c9fe575b10" modify_external_catalogs: true
I20240507 08:11:56.182294 5841 ts_tablet_manager.cc:1875] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.182399 5839 ts_tablet_manager.cc:1398] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:11:56.182943 5839 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.186221 5843 raft_consensus.cc:348] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.186479 5843 raft_consensus.cc:374] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.186569 5843 raft_consensus.cc:729] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:11:56.125661 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a7df45c39d1b490db97c72a0eb6cf3ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7df45c39d1b490db97c72a0eb6cf3ba already in progress: creating tablet
W20240507 08:11:56.187953 5248 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 62318.5us
I20240507 08:11:56.189915 5940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240507 08:11:56.190222 5841 ts_tablet_manager.cc:1888] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.190434 5841 log.cc:1196] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f29568ebe4774662a8b49b7829cf30e7
I20240507 08:11:56.190847 5841 ts_tablet_manager.cc:1909] T f29568ebe4774662a8b49b7829cf30e7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.191919 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f29568ebe4774662a8b49b7829cf30e7 (table default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]) successfully deleted
W20240507 08:11:56.192695 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a7df45c39d1b490db97c72a0eb6cf3ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7df45c39d1b490db97c72a0eb6cf3ba already in progress: creating tablet
I20240507 08:11:56.190243 5843 consensus_queue.cc:260] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.194559 5843 ts_tablet_manager.cc:1398] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.021s user 0.001s sys 0.002s
I20240507 08:11:56.197314 5843 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.199456 5843 tablet_bootstrap.cc:654] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.201771 5840 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.202023 5840 ts_tablet_manager.cc:1367] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.146s user 0.041s sys 0.034s
I20240507 08:11:56.202895 6187 tablet.cc:1717] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Skipping requested alter to schema version 3, tablet already version 3
I20240507 08:11:56.203016 5840 raft_consensus.cc:348] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.203227 5840 raft_consensus.cc:374] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.203316 5840 raft_consensus.cc:729] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.203576 5840 consensus_queue.cc:260] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.205948 5839 tablet_bootstrap.cc:654] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.206521 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
I20240507 08:11:56.207342 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2603fc71c01940698e55cf91cc063fab" } new_table_name: "default.Table_90a87f2fadb249189e0eb022d3730409" modify_external_catalogs: true
I20240507 08:11:56.207892 5840 ts_tablet_manager.cc:1398] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:56.209079 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39" } modify_external_catalogs: true
I20240507 08:11:56.209276 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39" } modify_external_catalogs: true
I20240507 08:11:56.209596 5838 ts_tablet_manager.cc:1888] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.209712 5838 log.cc:1196] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e0f3ee2b04de4cdf8a621b39f1476577
I20240507 08:11:56.210059 5838 ts_tablet_manager.cc:1909] T e0f3ee2b04de4cdf8a621b39f1476577 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.211715 5840 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.212581 5843 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.212736 5843 ts_tablet_manager.cc:1367] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20240507 08:11:56.212952 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e0f3ee2b04de4cdf8a621b39f1476577 (table default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]) successfully deleted
I20240507 08:11:56.213289 5840 tablet_bootstrap.cc:654] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.213400 5843 raft_consensus.cc:348] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.213624 5843 raft_consensus.cc:374] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.213712 5843 raft_consensus.cc:729] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.213927 5843 consensus_queue.cc:260] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.214490 5843 ts_tablet_manager.cc:1398] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:11:56.222043 5843 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.223721 5843 tablet_bootstrap.cc:654] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.233776 5843 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.235222 5843 ts_tablet_manager.cc:1367] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20240507 08:11:56.236162 5843 raft_consensus.cc:348] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.236449 5843 raft_consensus.cc:374] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.236541 5843 raft_consensus.cc:729] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.236796 5843 consensus_queue.cc:260] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.242460 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fea6e34b279d43b2918afeac8c53ceec" } modify_external_catalogs: true
I20240507 08:11:56.242658 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fea6e34b279d43b2918afeac8c53ceec" } modify_external_catalogs: true
I20240507 08:11:56.243520 5843 ts_tablet_manager.cc:1398] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:11:56.246789 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 0118ef28b24641d68cf24ce9df966243 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:56.247481 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_94b53721a4784e1b94415eb370b8dd9c [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 1 got 0
I20240507 08:11:56.247664 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]). Expected version 3 got 0
W20240507 08:11:56.248770 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4c064b969844d938fddfe7f043c0b92 consensus state has no leader
I20240507 08:11:56.249079 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 7027b4cefa4d41659f819ee4326470c4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:56.254909 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba24c078af9472da06793c58f147290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.256148 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9ba24c078af9472da06793c58f147290 because the tablet was not found. No further retry: Not found: Tablet not found: 9ba24c078af9472da06793c58f147290
I20240507 08:11:56.259526 5839 tablet_bootstrap.cc:492] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.259701 5839 ts_tablet_manager.cc:1367] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.021s
I20240507 08:11:56.260298 5843 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.260355 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.260329 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.260973 5841 tablet_replica.cc:330] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.261163 5841 raft_consensus.cc:2229] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.261482 5841 raft_consensus.cc:2258] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.262923 5841 ts_tablet_manager.cc:1875] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.263607 5843 tablet_bootstrap.cc:654] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.271392 5841 ts_tablet_manager.cc:1888] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.271612 5841 log.cc:1196] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/344bf4ab39f74accb383d695578de604
I20240507 08:11:56.272014 5841 ts_tablet_manager.cc:1909] T 344bf4ab39f74accb383d695578de604 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.273212 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 344bf4ab39f74accb383d695578de604 (table default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]) successfully deleted
I20240507 08:11:56.273378 5839 raft_consensus.cc:348] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.273648 5839 raft_consensus.cc:374] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.274068 5841 tablet_replica.cc:330] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.274366 5841 raft_consensus.cc:2229] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.274566 5841 raft_consensus.cc:2258] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.273768 5839 raft_consensus.cc:729] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.275569 5839 consensus_queue.cc:260] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.277932 5839 ts_tablet_manager.cc:1398] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240507 08:11:56.279611 5841 ts_tablet_manager.cc:1875] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.280972 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_086d5c28a17243c9991e0add9910274a [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 2 got 0
I20240507 08:11:56.286502 5839 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.288398 5839 tablet_bootstrap.cc:654] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.289292 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 831153d2b89e43f8b08b086f7a8f3a54 consensus state has no leader
I20240507 08:11:56.289635 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet b22334378eec4ab3885d0f9d2d812a3e from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:56.290133 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet b7947ca2d8324dbb9c909ec50852fed8 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:56.292474 5841 ts_tablet_manager.cc:1888] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.292626 5841 log.cc:1196] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:56.292971 5841 ts_tablet_manager.cc:1909] T cb7acbdb357445adb9ce69103c782a7f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.294327 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet cb7acbdb357445adb9ce69103c782a7f (table default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]) successfully deleted
I20240507 08:11:56.294425 5843 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.294612 5843 ts_tablet_manager.cc:1367] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.047s user 0.001s sys 0.013s
I20240507 08:11:56.295184 5843 raft_consensus.cc:348] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.295353 5843 raft_consensus.cc:374] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.295423 5843 raft_consensus.cc:729] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.295656 5843 consensus_queue.cc:260] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.296123 5843 ts_tablet_manager.cc:1398] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240507 08:11:56.296526 5843 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.297677 5839 tablet_bootstrap.cc:492] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.298177 5839 ts_tablet_manager.cc:1367] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20240507 08:11:56.298305 5843 tablet_bootstrap.cc:654] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.299096 5839 raft_consensus.cc:348] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.299396 5839 raft_consensus.cc:374] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.299501 5839 raft_consensus.cc:729] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.299865 5839 consensus_queue.cc:260] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.301910 5839 ts_tablet_manager.cc:1398] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:56.303577 5839 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.307353 5839 tablet_bootstrap.cc:654] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.316987 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 565ac6091f5e48b1b024094d5560e275 consensus state has no leader
I20240507 08:11:56.317399 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cebf0395fd3b4cf1a49263069e98ae91" } modify_external_catalogs: true
I20240507 08:11:56.317625 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cebf0395fd3b4cf1a49263069e98ae91" } modify_external_catalogs: true
I20240507 08:11:56.321390 6087 raft_consensus.cc:482] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.321566 6087 raft_consensus.cc:504] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.322346 6087 leader_election.cc:288] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:56.323076 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e5439844f384513b4719cfe4f4e5d9b has no consensus state
I20240507 08:11:56.323493 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5e90ac34a69740dca680fc75e5738dcd" } new_table_name: "default.Table_19b106706150440f8c07e95b72d7bde8" modify_external_catalogs: true
I20240507 08:11:56.323581 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "344bf4ab39f74accb383d695578de604" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.324716 5654 leader_election.cc:341] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:56.325273 5656 leader_election.cc:341] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 344bf4ab39f74accb383d695578de604
I20240507 08:11:56.325390 5656 leader_election.cc:302] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.325917 6087 raft_consensus.cc:2726] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.327021 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cd4379b6fded4a52843f55f27ae67121
I20240507 08:11:56.327864 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_81657f34e12b4ca0b2fd8ee26ab0e663" } modify_external_catalogs: true
I20240507 08:11:56.328063 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_81657f34e12b4ca0b2fd8ee26ab0e663" } modify_external_catalogs: true
I20240507 08:11:56.328573 5264 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807] alter complete (version 3)
I20240507 08:11:56.329229 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_53e17078e5714619b86165f5e3f636ff" } modify_external_catalogs: true
I20240507 08:11:56.329425 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_53e17078e5714619b86165f5e3f636ff" } modify_external_catalogs: true
I20240507 08:11:56.330632 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c2d5a28991f9466486b34132cbd61131"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.331175 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2d5a28991f9466486b34132cbd61131 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.332646 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.333464 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cd4379b6fded4a52843f55f27ae67121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4379b6fded4a52843f55f27ae67121 already in progress: creating tablet
I20240507 08:11:56.335695 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.335996 5837 tablet_replica.cc:330] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:56.336228 5837 raft_consensus.cc:2229] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.336421 5837 raft_consensus.cc:2258] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.337380 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.338130 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.338407 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 4867e1cca1634b4ab6e36f9ed3204e0a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.338909 5838 tablet_replica.cc:330] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.339104 5838 raft_consensus.cc:2229] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.339262 5838 raft_consensus.cc:2258] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.340302 5838 ts_tablet_manager.cc:1875] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.340405 5837 ts_tablet_manager.cc:1875] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:56.341936 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cd4379b6fded4a52843f55f27ae67121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4379b6fded4a52843f55f27ae67121 already in progress: creating tablet
I20240507 08:11:56.342288 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.342597 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 812dd1ebd42f4e929838a0535c22ad69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.343482 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.343716 5841 tablet_replica.cc:330] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.343905 5841 raft_consensus.cc:2229] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.344070 5841 raft_consensus.cc:2258] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.344373 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.344934 5841 ts_tablet_manager.cc:1875] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.345592 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.346163 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet ebc6cc72bb7e4721a8ab559102273db5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.346812 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a778d8595fc7496bace57c63194455fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.347445 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a778d8595fc7496bace57c63194455fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.347723 5837 ts_tablet_manager.cc:1888] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.348362 5837 log.cc:1196] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/344bf4ab39f74accb383d695578de604
I20240507 08:11:56.348932 5837 ts_tablet_manager.cc:1909] T 344bf4ab39f74accb383d695578de604 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:56.350122 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 344bf4ab39f74accb383d695578de604 (table default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]) successfully deleted
W20240507 08:11:56.350829 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet cd4379b6fded4a52843f55f27ae67121 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4379b6fded4a52843f55f27ae67121 already in progress: creating tablet
I20240507 08:11:56.351190 5837 tablet_replica.cc:330] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:56.351444 5837 raft_consensus.cc:2229] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.351678 5837 raft_consensus.cc:2258] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.352095 5841 ts_tablet_manager.cc:1888] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.352227 5841 log.cc:1196] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/67c8624285d946e597227430a5714e2f
I20240507 08:11:56.352641 5841 ts_tablet_manager.cc:1909] T 67c8624285d946e597227430a5714e2f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.352658 5837 ts_tablet_manager.cc:1875] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.353699 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 67c8624285d946e597227430a5714e2f (table default.Table_97f2cc68bfaf4e558499da922503bdb1 [id=bce0bd66e7b74db7934937410c81b20b]) successfully deleted
I20240507 08:11:56.354055 5838 ts_tablet_manager.cc:1888] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.354225 5838 log.cc:1196] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/97a43c27b30a4c3cbfd80dce5f28d7c7
I20240507 08:11:56.354372 5841 tablet_replica.cc:330] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.354611 5838 ts_tablet_manager.cc:1909] T 97a43c27b30a4c3cbfd80dce5f28d7c7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.354744 5841 raft_consensus.cc:2229] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.354959 5841 raft_consensus.cc:2258] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.355988 5838 tablet_replica.cc:330] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.356034 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 97a43c27b30a4c3cbfd80dce5f28d7c7 (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
I20240507 08:11:56.356212 5838 raft_consensus.cc:2229] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.356436 5838 raft_consensus.cc:2258] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.357251 5841 ts_tablet_manager.cc:1875] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.358044 5838 ts_tablet_manager.cc:1875] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:56.363842 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5782cfa5eca74c19a2e724a9124597f2 consensus state has no leader
I20240507 08:11:56.369951 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0ab8cb559bbe4154a11058e94d2ad349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.370483 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ab8cb559bbe4154a11058e94d2ad349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.375428 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet b2df872d04b94c30872b03367df3cc31 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:56.377112 5840 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.377316 5840 ts_tablet_manager.cc:1367] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.166s user 0.030s sys 0.049s
I20240507 08:11:56.377980 5843 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.378168 5843 ts_tablet_manager.cc:1367] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.082s user 0.035s sys 0.022s
I20240507 08:11:56.378345 5840 raft_consensus.cc:348] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.378602 5840 raft_consensus.cc:374] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.378710 5840 raft_consensus.cc:729] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.378875 5841 ts_tablet_manager.cc:1888] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.379024 5841 log.cc:1196] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/ea17889619fa420fbeacf1b41594c598
I20240507 08:11:56.379067 5840 consensus_queue.cc:260] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.379142 5843 raft_consensus.cc:348] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.379391 5843 raft_consensus.cc:374] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.379699 5843 raft_consensus.cc:729] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.380834 5843 consensus_queue.cc:260] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.381556 5843 ts_tablet_manager.cc:1398] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:56.381934 5841 ts_tablet_manager.cc:1909] T ea17889619fa420fbeacf1b41594c598 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.382020 5843 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.383284 5841 tablet_replica.cc:330] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.383428 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet ea17889619fa420fbeacf1b41594c598 (table default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]) successfully deleted
I20240507 08:11:56.383481 5841 raft_consensus.cc:2229] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.383649 5841 raft_consensus.cc:2258] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.384027 6087 raft_consensus.cc:482] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.384164 6087 raft_consensus.cc:504] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.384821 6087 leader_election.cc:288] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:56.384939 5843 tablet_bootstrap.cc:654] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.385563 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972bc0a5ba924e0fb946cdacd9575c1e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.386512 5654 leader_election.cc:341] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.387277 5841 ts_tablet_manager.cc:1875] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.387238 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972bc0a5ba924e0fb946cdacd9575c1e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:56.387581 5443 raft_consensus.cc:2454] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:11:56.387989 5656 leader_election.cc:302] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.388300 6087 raft_consensus.cc:482] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.388450 6087 raft_consensus.cc:504] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.388990 6087 leader_election.cc:288] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:56.389143 6087 raft_consensus.cc:2781] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:56.389215 6087 raft_consensus.cc:482] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:56.389307 6087 raft_consensus.cc:3037] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.391968 6087 raft_consensus.cc:504] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.392495 6087 leader_election.cc:288] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:56.393237 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972bc0a5ba924e0fb946cdacd9575c1e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:56.393683 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67c8624285d946e597227430a5714e2f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.394138 5654 leader_election.cc:341] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:56.394340 5654 leader_election.cc:341] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.395511 5840 ts_tablet_manager.cc:1398] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240507 08:11:56.396237 5841 ts_tablet_manager.cc:1888] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.396396 5841 log.cc:1196] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5ca8fc2c7f3849c380fea56862ffc26e
I20240507 08:11:56.396816 5841 ts_tablet_manager.cc:1909] T 5ca8fc2c7f3849c380fea56862ffc26e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.397353 5840 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.398471 5843 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.398656 5843 ts_tablet_manager.cc:1367] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.005s
I20240507 08:11:56.399719 5843 raft_consensus.cc:348] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.400012 5843 raft_consensus.cc:374] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.400056 5840 tablet_bootstrap.cc:654] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.400112 5843 raft_consensus.cc:729] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.400596 5843 consensus_queue.cc:260] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.401476 5843 ts_tablet_manager.cc:1398] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240507 08:11:56.401777 5843 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.401885 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.403323 5843 tablet_bootstrap.cc:654] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.404742 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5ca8fc2c7f3849c380fea56862ffc26e (table default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]) successfully deleted
I20240507 08:11:56.405048 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_615d90859dd74c9aa813e233b63dd68e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.405936 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_615d90859dd74c9aa813e233b63dd68e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.407441 5837 ts_tablet_manager.cc:1888] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.407575 5837 log.cc:1196] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:56.414731 5838 ts_tablet_manager.cc:1888] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.414963 5838 log.cc:1196] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/812dd1ebd42f4e929838a0535c22ad69
I20240507 08:11:56.415344 5838 ts_tablet_manager.cc:1909] T 812dd1ebd42f4e929838a0535c22ad69 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.416422 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 812dd1ebd42f4e929838a0535c22ad69 (table default.Table_f29e8299e1fc4d2ba2b6cd3036641274 [id=a764e2dd5a9841fe9ebc72c179b7858e]) successfully deleted
I20240507 08:11:56.418112 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "972bc0a5ba924e0fb946cdacd9575c1e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:11:56.418391 5443 raft_consensus.cc:3037] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:56.421535 5656 leader_election.cc:341] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 67c8624285d946e597227430a5714e2f
I20240507 08:11:56.421842 5656 leader_election.cc:302] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.423940 5443 raft_consensus.cc:2454] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:11:56.424527 5837 ts_tablet_manager.cc:1909] T cb7acbdb357445adb9ce69103c782a7f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:56.424559 5656 leader_election.cc:302] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.424928 6087 raft_consensus.cc:2726] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.425215 6087 raft_consensus.cc:2781] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:56.425396 6087 raft_consensus.cc:686] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:11:56.425860 6087 consensus_queue.cc:237] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.426160 5843 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.426353 5843 ts_tablet_manager.cc:1367] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.009s
I20240507 08:11:56.427379 5843 raft_consensus.cc:348] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.427701 5843 raft_consensus.cc:374] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.427772 5843 raft_consensus.cc:729] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.428005 5843 consensus_queue.cc:260] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.428166 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.428527 5843 ts_tablet_manager.cc:1398] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240507 08:11:56.428774 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.428954 5843 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.430970 5843 tablet_bootstrap.cc:654] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.432487 5839 tablet_bootstrap.cc:492] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.432695 5839 ts_tablet_manager.cc:1367] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.129s user 0.024s sys 0.023s
I20240507 08:11:56.433740 5839 raft_consensus.cc:348] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.434069 5839 raft_consensus.cc:374] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.434190 5839 raft_consensus.cc:729] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.434468 5839 consensus_queue.cc:260] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.435621 5839 ts_tablet_manager.cc:1398] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:56.437219 5839 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.438588 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet cb7acbdb357445adb9ce69103c782a7f (table default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]) successfully deleted
I20240507 08:11:56.439033 5837 tablet_replica.cc:330] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:56.439317 5837 raft_consensus.cc:2229] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.439594 5843 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.439713 5839 tablet_bootstrap.cc:654] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.439764 5843 ts_tablet_manager.cc:1367] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240507 08:11:56.440559 5843 raft_consensus.cc:348] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.440748 5843 raft_consensus.cc:374] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.440830 5843 raft_consensus.cc:729] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.441077 5843 consensus_queue.cc:260] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.442778 5843 ts_tablet_manager.cc:1398] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:56.443034 5843 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.445156 5843 tablet_bootstrap.cc:654] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.450722 5840 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.450995 5840 ts_tablet_manager.cc:1367] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.004s
I20240507 08:11:56.451491 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet c4899c088531480ea15364af94595ee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.451885 5840 raft_consensus.cc:348] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.452114 5840 raft_consensus.cc:374] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.452191 5840 raft_consensus.cc:729] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.452445 5840 consensus_queue.cc:260] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.452941 5840 ts_tablet_manager.cc:1398] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:56.453240 5840 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.454180 5838 tablet_replica.cc:330] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.456104 5838 raft_consensus.cc:2229] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.456298 5838 raft_consensus.cc:2258] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.456454 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4fdb46838aab420bb1038ea941103c52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.456872 6073 raft_consensus.cc:482] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.456997 6073 raft_consensus.cc:504] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:56.457206 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fdb46838aab420bb1038ea941103c52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.457298 5838 ts_tablet_manager.cc:1875] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.457680 6073 leader_election.cc:288] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:56.458808 5520 leader_election.cc:341] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: c4899c088531480ea15364af94595ee5
W20240507 08:11:56.459576 5517 leader_election.cc:341] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: c4899c088531480ea15364af94595ee5
I20240507 08:11:56.459719 5517 leader_election.cc:302] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:56.460204 6073 raft_consensus.cc:2726] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.463560 5839 tablet_bootstrap.cc:492] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.465404 5839 ts_tablet_manager.cc:1367] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.000s
I20240507 08:11:56.466513 5839 raft_consensus.cc:348] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.466835 5839 raft_consensus.cc:374] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.466934 5839 raft_consensus.cc:729] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.467253 5839 consensus_queue.cc:260] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.469792 5839 ts_tablet_manager.cc:1398] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:56.470460 5839 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.473752 5840 tablet_bootstrap.cc:654] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.456554 5837 raft_consensus.cc:2258] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.474901 5837 ts_tablet_manager.cc:1875] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.486555 5839 tablet_bootstrap.cc:654] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.490878 5843 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.491123 5843 ts_tablet_manager.cc:1367] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.048s user 0.017s sys 0.024s
I20240507 08:11:56.492130 5843 raft_consensus.cc:348] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.492363 5843 raft_consensus.cc:374] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.492436 5843 raft_consensus.cc:729] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.493204 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9429e5cc274198974b27c40c7e4e37 (DEFAULT_TABLE table=default.Table_f2d4ce8cce7c491981416e3efa5a6d11 [id=9fbcfe2b38f0426dbcbb0fb437264080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.493417 5843 consensus_queue.cc:260] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.493599 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9429e5cc274198974b27c40c7e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.494009 5843 ts_tablet_manager.cc:1398] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:56.494338 5843 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.495992 5843 tablet_bootstrap.cc:654] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.498064 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 4e5439844f384513b4719cfe4f4e5d9b (table default.Table_90a87f2fadb249189e0eb022d3730409 [id=64f54c3a3fba416f8810335c6f651ced], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4e5439844f384513b4719cfe4f4e5d9b consensus state has no leader
I20240507 08:11:56.504096 5843 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.504213 5838 ts_tablet_manager.cc:1888] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.504259 5843 ts_tablet_manager.cc:1367] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20240507 08:11:56.504374 5838 log.cc:1196] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/ebc6cc72bb7e4721a8ab559102273db5
I20240507 08:11:56.504415 5837 ts_tablet_manager.cc:1888] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.504583 5837 log.cc:1196] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/67c8624285d946e597227430a5714e2f
I20240507 08:11:56.504794 5838 ts_tablet_manager.cc:1909] T ebc6cc72bb7e4721a8ab559102273db5 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.504947 5837 ts_tablet_manager.cc:1909] T 67c8624285d946e597227430a5714e2f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:56.505102 5843 raft_consensus.cc:348] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.505339 5843 raft_consensus.cc:374] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.505447 5843 raft_consensus.cc:729] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.505507 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet ebc6cc72bb7e4721a8ab559102273db5 (table default.Table_fcab59c9e0064c3487c28efc45b90928 [id=a9c2a5bfc5ce4b059b2a3b2fc568a68a]) successfully deleted
I20240507 08:11:56.505699 5843 consensus_queue.cc:260] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.506449 5843 ts_tablet_manager.cc:1398] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:56.506832 5843 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.507524 5838 tablet_replica.cc:330] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.507746 5838 raft_consensus.cc:2229] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.507948 5838 raft_consensus.cc:2258] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.508586 5843 tablet_bootstrap.cc:654] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.508767 5839 tablet_bootstrap.cc:492] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.509346 5839 ts_tablet_manager.cc:1367] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
I20240507 08:11:56.510318 5839 raft_consensus.cc:348] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.510529 5839 raft_consensus.cc:374] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.510684 5839 raft_consensus.cc:729] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.510887 5843 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.510967 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet add94bcf0c844cb78059b7240c7e28c0 with delete_type TABLET_DATA_DELETED (replaced by tablet b7947ca2d8324dbb9c909ec50852fed8) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.511035 5843 ts_tablet_manager.cc:1367] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240507 08:11:56.511222 5839 consensus_queue.cc:260] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.511818 5843 raft_consensus.cc:348] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.512012 5843 raft_consensus.cc:374] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.512085 5843 raft_consensus.cc:729] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.512154 5839 ts_tablet_manager.cc:1398] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:56.512308 5843 consensus_queue.cc:260] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.512936 5843 ts_tablet_manager.cc:1398] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:11:56.513204 5839 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.513298 5843 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.513300 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0118ef28b24641d68cf24ce9df966243
I20240507 08:11:56.513844 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 0118ef28b24641d68cf24ce9df966243 deleted and replaced by tablet 3fce98f472ef46589dbea6b6aefa6572
I20240507 08:11:56.514039 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.514855 5838 ts_tablet_manager.cc:1875] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.514923 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:11:56.514989 5843 tablet_bootstrap.cc:654] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.515558 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.516705 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ed36193e2b664d80b2a509e4f94623bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.517239 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed36193e2b664d80b2a509e4f94623bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.517395 5839 tablet_bootstrap.cc:654] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.518258 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.518530 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.519089 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0118ef28b24641d68cf24ce9df966243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0118ef28b24641d68cf24ce9df966243 already in progress: creating tablet
W20240507 08:11:56.519357 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 33a425a90f0449bcaaacb4c3f653d26c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33a425a90f0449bcaaacb4c3f653d26c already in progress: creating tablet
I20240507 08:11:56.521528 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.521795 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.527124 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 67c8624285d946e597227430a5714e2f (table default.Table_97f2cc68bfaf4e558499da922503bdb1 [id=bce0bd66e7b74db7934937410c81b20b]) successfully deleted
I20240507 08:11:56.527498 5838 ts_tablet_manager.cc:1888] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.527673 5838 log.cc:1196] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c4899c088531480ea15364af94595ee5
I20240507 08:11:56.528192 5838 ts_tablet_manager.cc:1909] T c4899c088531480ea15364af94595ee5 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.528647 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c04dfdb03826409ca727e76221578ea3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.529006 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c4899c088531480ea15364af94595ee5 (table default.Table_8ad438587c5c48c78d151533b16a55e3 [id=307b8b3612c1470d916ad0f37c986171]) successfully deleted
W20240507 08:11:56.529332 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0118ef28b24641d68cf24ce9df966243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0118ef28b24641d68cf24ce9df966243 already in progress: creating tablet
I20240507 08:11:56.529445 5838 tablet_replica.cc:330] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:11:56.529614 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c04dfdb03826409ca727e76221578ea3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.530251 5839 tablet_bootstrap.cc:492] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:11:56.530335 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 33a425a90f0449bcaaacb4c3f653d26c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33a425a90f0449bcaaacb4c3f653d26c already in progress: creating tablet
I20240507 08:11:56.530472 5839 ts_tablet_manager.cc:1367] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.003s
I20240507 08:11:56.531512 5839 raft_consensus.cc:348] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.531827 5839 raft_consensus.cc:374] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.531910 5839 raft_consensus.cc:729] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.532261 5839 consensus_queue.cc:260] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.535511 5839 ts_tablet_manager.cc:1398] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240507 08:11:56.535856 5838 raft_consensus.cc:2229] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.536082 5838 raft_consensus.cc:2258] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.537043 5838 ts_tablet_manager.cc:1875] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.537943 5839 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.539685 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6f6670e99e4a4442801d42950eb7ab79
I20240507 08:11:56.540452 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.540683 5839 tablet_bootstrap.cc:654] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.541249 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.542028 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6f6670e99e4a4442801d42950eb7ab79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f6670e99e4a4442801d42950eb7ab79 already in progress: creating tablet
I20240507 08:11:56.543150 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:56.543869 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6f6670e99e4a4442801d42950eb7ab79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f6670e99e4a4442801d42950eb7ab79 already in progress: creating tablet
I20240507 08:11:56.547231 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9429e5cc274198974b27c40c7e4e37 (DEFAULT_TABLE table=default.Table_f2d4ce8cce7c491981416e3efa5a6d11 [id=9fbcfe2b38f0426dbcbb0fb437264080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.547665 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9429e5cc274198974b27c40c7e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.552726 5838 ts_tablet_manager.cc:1888] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.553066 5838 log.cc:1196] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/add94bcf0c844cb78059b7240c7e28c0
I20240507 08:11:56.558140 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 3b9429e5cc274198974b27c40c7e4e37 (DEFAULT_TABLE table=default.Table_f2d4ce8cce7c491981416e3efa5a6d11 [id=9fbcfe2b38f0426dbcbb0fb437264080]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.558687 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b9429e5cc274198974b27c40c7e4e37. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.559006 5839 tablet_bootstrap.cc:492] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.559278 5839 ts_tablet_manager.cc:1367] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20240507 08:11:56.560290 5839 raft_consensus.cc:348] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.560564 5839 raft_consensus.cc:374] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.560739 5839 raft_consensus.cc:729] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.561051 5839 consensus_queue.cc:260] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.562243 5839 ts_tablet_manager.cc:1398] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:56.562664 5839 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.564040 5995 consensus_queue.cc:1035] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:56.553563 5838 ts_tablet_manager.cc:1909] T add94bcf0c844cb78059b7240c7e28c0 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.566138 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet add94bcf0c844cb78059b7240c7e28c0 (table default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:11:56.566433 5839 tablet_bootstrap.cc:654] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.568115 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0118ef28b24641d68cf24ce9df966243 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0118ef28b24641d68cf24ce9df966243 already in progress: creating tablet
W20240507 08:11:56.568836 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 33a425a90f0449bcaaacb4c3f653d26c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33a425a90f0449bcaaacb4c3f653d26c already in progress: creating tablet
W20240507 08:11:56.569612 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6f6670e99e4a4442801d42950eb7ab79 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f6670e99e4a4442801d42950eb7ab79 already in progress: creating tablet
I20240507 08:11:56.570472 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5228590c8e7d4c34b3acab9eb16665d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.570391 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_38e25aba16ae4bcab73abbb11562bfd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.571156 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38e25aba16ae4bcab73abbb11562bfd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.571676 5838 tablet_replica.cc:330] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.571925 5838 raft_consensus.cc:2229] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.572141 5838 raft_consensus.cc:2258] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.572860 5838 ts_tablet_manager.cc:1875] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.574039 5995 raft_consensus.cc:482] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.574211 5995 raft_consensus.cc:504] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.574895 5995 leader_election.cc:288] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:56.575680 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.576381 5378 leader_election.cc:341] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.577139 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:56.577383 5718 raft_consensus.cc:2454] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:56.578159 5377 leader_election.cc:302] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.578655 5995 raft_consensus.cc:2781] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:56.578754 5995 raft_consensus.cc:482] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:56.578855 5995 raft_consensus.cc:3037] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.580873 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7027b4cefa4d41659f819ee4326470c4
I20240507 08:11:56.581393 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 7027b4cefa4d41659f819ee4326470c4 deleted and replaced by tablet 66247e08dfdf47f69ba24b074c8080f0
I20240507 08:11:56.581650 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.582003 5995 raft_consensus.cc:504] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.582665 5839 tablet_bootstrap.cc:492] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.582880 5839 ts_tablet_manager.cc:1367] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20240507 08:11:56.583446 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:56.583446 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3740c610e0f047f4b335d6e49868a581" } new_table_name: "default.Table_0f7557ddeec745f59ac90d4085a9015c" modify_external_catalogs: true
I20240507 08:11:56.583529 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5cab176c4bd34f38b14ac096f79b912e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:50 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.583639 5718 raft_consensus.cc:3037] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.583745 5839 raft_consensus.cc:348] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.584054 5839 raft_consensus.cc:374] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.584134 5839 raft_consensus.cc:729] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.584467 5839 consensus_queue.cc:260] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.585736 6155 raft_consensus.cc:482] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.585896 6155 raft_consensus.cc:504] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.586576 6155 leader_election.cc:288] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.586980 5839 ts_tablet_manager.cc:1398] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240507 08:11:56.587157 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:56.587394 5839 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:56.587663 5378 leader_election.cc:341] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.587641 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.588353 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7027b4cefa4d41659f819ee4326470c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7027b4cefa4d41659f819ee4326470c4 already in progress: creating tablet
I20240507 08:11:56.589746 5555 tablet_service.cc:1467] Processing CreateTablet for tablet e14487e625374060b118c9cce4500b3f (DEFAULT_TABLE table=default.Table_5efeda526693400790a6dfb0d118fac0 [id=8adafd6d08374df7a4f0fa1fd90991fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.590166 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14487e625374060b118c9cce4500b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.590929 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3bdb777b974524b5673a3f02883107 (DEFAULT_TABLE table=default.Table_18d4760f5bf94ab69b1ce6418ad9507c [id=98125a41d31245e6975dc9393d2faef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.591326 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3bdb777b974524b5673a3f02883107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.594753 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b2df872d04b94c30872b03367df3cc31
I20240507 08:11:56.595243 5261 catalog_manager.cc:6554] ReplaceTablet: tablet b2df872d04b94c30872b03367df3cc31 deleted and replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f
I20240507 08:11:56.595515 5995 leader_election.cc:288] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:56.595763 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.596202 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:56.596688 5718 raft_consensus.cc:2454] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:56.596988 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7c0ab3c1b77745b796fe0a735b7bc292"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:56.597108 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:56.597328 5717 raft_consensus.cc:2454] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:56.597757 5377 leader_election.cc:302] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:56.598160 5995 raft_consensus.cc:2781] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:56.598217 5377 leader_election.cc:302] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.598899 6155 raft_consensus.cc:2781] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:56.598949 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.599004 6155 raft_consensus.cc:482] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:56.599112 6155 raft_consensus.cc:3037] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.599699 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 972bc0a5ba924e0fb946cdacd9575c1e
I20240507 08:11:56.600430 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_992460c354bb4a4c9b51dcf2db8657ae" } new_table_name: "default.Table_8ebdb9219e974871b5a4f78b6abfd78f" modify_external_catalogs: true
I20240507 08:11:56.601998 5995 raft_consensus.cc:686] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:56.602267 5995 consensus_queue.cc:237] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.602800 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a94e166f91d741cd9c5e85ea5efb8e3f" } new_table_name: "default.Table_aa2b432cc7ef4604b737ade9696fe60f" modify_external_catalogs: true
I20240507 08:11:56.603598 6155 raft_consensus.cc:504] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.604354 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3b339252e2644a438c81a546b3cdb046" } new_table_name: "default.Table_0856317e9911474aa5861773d7185325" modify_external_catalogs: true
I20240507 08:11:56.605248 6155 leader_election.cc:288] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.605674 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.605937 5841 tablet_replica.cc:330] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:56.605801 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:56.606101 5717 raft_consensus.cc:3037] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.606137 5841 raft_consensus.cc:2229] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.606292 5841 raft_consensus.cc:2258] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:56.606612 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b2df872d04b94c30872b03367df3cc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2df872d04b94c30872b03367df3cc31 already in progress: creating tablet
I20240507 08:11:56.607249 5841 ts_tablet_manager.cc:1875] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.609007 5683 tablet_service.cc:1467] Processing CreateTablet for tablet e14487e625374060b118c9cce4500b3f (DEFAULT_TABLE table=default.Table_5efeda526693400790a6dfb0d118fac0 [id=8adafd6d08374df7a4f0fa1fd90991fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.609447 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14487e625374060b118c9cce4500b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.609862 5717 raft_consensus.cc:2454] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:56.610162 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:56.610580 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7027b4cefa4d41659f819ee4326470c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7027b4cefa4d41659f819ee4326470c4 already in progress: creating tablet
W20240507 08:11:56.610836 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b2df872d04b94c30872b03367df3cc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2df872d04b94c30872b03367df3cc31 already in progress: creating tablet
I20240507 08:11:56.610843 5377 leader_election.cc:302] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:56.611176 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.611181 5995 raft_consensus.cc:2781] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:56.611428 5837 tablet_replica.cc:330] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:56.611497 5995 raft_consensus.cc:686] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:56.611624 5837 raft_consensus.cc:2229] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:56.611871 5837 pending_rounds.cc:62] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:11:56.611933 5837 pending_rounds.cc:68] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924739283927040 op_type: NO_OP noop_request { }
I20240507 08:11:56.611864 5995 consensus_queue.cc:237] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.612064 5837 raft_consensus.cc:2866] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:11:56.612255 5837 raft_consensus.cc:2258] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.615633 5839 tablet_bootstrap.cc:654] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.615644 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c0ab3c1b77745b796fe0a735b7bc292 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.622785 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:56.624770 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0402c6efc8412aac0d21a545944073" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:56.625011 5582 raft_consensus.cc:3037] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:11:56.625458 5378 leader_election.cc:341] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.626765 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e14487e625374060b118c9cce4500b3f (DEFAULT_TABLE table=default.Table_5efeda526693400790a6dfb0d118fac0 [id=8adafd6d08374df7a4f0fa1fd90991fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.629148 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e14487e625374060b118c9cce4500b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.629248 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.629839 5837 ts_tablet_manager.cc:1875] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.628314 5841 ts_tablet_manager.cc:1888] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.632079 5841 log.cc:1196] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/972bc0a5ba924e0fb946cdacd9575c1e
I20240507 08:11:56.632696 5841 ts_tablet_manager.cc:1909] T 972bc0a5ba924e0fb946cdacd9575c1e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:56.634402 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 972bc0a5ba924e0fb946cdacd9575c1e (table default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]) successfully deleted
I20240507 08:11:56.634756 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.635047 5838 ts_tablet_manager.cc:1888] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.635190 5838 log.cc:1196] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5228590c8e7d4c34b3acab9eb16665d3
I20240507 08:11:56.635674 5838 ts_tablet_manager.cc:1909] T 5228590c8e7d4c34b3acab9eb16665d3 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.636399 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5228590c8e7d4c34b3acab9eb16665d3 (table default.Table_eff2f49d93764317a595c308278c0732 [id=2b718529a9e7440d831c3d60a8c17ba4]) successfully deleted
I20240507 08:11:56.637014 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3bdb777b974524b5673a3f02883107 (DEFAULT_TABLE table=default.Table_18d4760f5bf94ab69b1ce6418ad9507c [id=98125a41d31245e6975dc9393d2faef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.637542 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3bdb777b974524b5673a3f02883107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.637631 5582 raft_consensus.cc:2454] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:11:56.640165 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7027b4cefa4d41659f819ee4326470c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7027b4cefa4d41659f819ee4326470c4 already in progress: creating tablet
I20240507 08:11:56.640575 5838 tablet_replica.cc:330] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:56.640839 5838 raft_consensus.cc:2229] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:56.641026 5838 raft_consensus.cc:2258] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:56.645692 5838 ts_tablet_manager.cc:1875] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:56.646422 5582 raft_consensus.cc:3037] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.650581 6226 consensus_queue.cc:1035] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:56.655272 5839 tablet_bootstrap.cc:492] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.655469 5839 ts_tablet_manager.cc:1367] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.068s user 0.002s sys 0.006s
I20240507 08:11:56.656471 5839 raft_consensus.cc:348] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.656730 5839 raft_consensus.cc:374] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.656848 5839 raft_consensus.cc:729] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.657138 5839 consensus_queue.cc:260] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.659150 5839 ts_tablet_manager.cc:1398] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:11:56.660642 5839 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.664364 5840 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.664568 5840 ts_tablet_manager.cc:1367] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.211s user 0.004s sys 0.009s
I20240507 08:11:56.665357 5840 raft_consensus.cc:348] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.665686 5840 raft_consensus.cc:374] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.665845 5840 raft_consensus.cc:729] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.666185 5840 consensus_queue.cc:260] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.666765 5839 tablet_bootstrap.cc:654] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.669732 5840 ts_tablet_manager.cc:1398] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:11:56.670873 5837 ts_tablet_manager.cc:1888] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:56.671469 5837 log.cc:1196] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/972bc0a5ba924e0fb946cdacd9575c1e
I20240507 08:11:56.673147 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3bdb777b974524b5673a3f02883107 (DEFAULT_TABLE table=default.Table_18d4760f5bf94ab69b1ce6418ad9507c [id=98125a41d31245e6975dc9393d2faef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.673739 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3bdb777b974524b5673a3f02883107. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.673744 5837 ts_tablet_manager.cc:1909] T 972bc0a5ba924e0fb946cdacd9575c1e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:56.677856 5839 tablet_bootstrap.cc:492] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.678064 5839 ts_tablet_manager.cc:1367] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20240507 08:11:56.678056 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 972bc0a5ba924e0fb946cdacd9575c1e (table default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]) successfully deleted
I20240507 08:11:56.682013 5839 raft_consensus.cc:348] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.682332 5839 raft_consensus.cc:374] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.682436 5839 raft_consensus.cc:729] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.683559 5838 ts_tablet_manager.cc:1888] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:56.683732 5838 log.cc:1196] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5cab176c4bd34f38b14ac096f79b912e
I20240507 08:11:56.684203 5838 ts_tablet_manager.cc:1909] T 5cab176c4bd34f38b14ac096f79b912e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:56.684844 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5cab176c4bd34f38b14ac096f79b912e (table default.Table_9aa4c69483c04eb1a1c0db1c0555cef1 [id=893f83870bb84ca8b0bfd97bed22cafa]) successfully deleted
I20240507 08:11:56.685989 5839 consensus_queue.cc:260] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.687536 5549 tablet_service.cc:1467] Processing CreateTablet for tablet b8c6f552832a45c4a2e28f40797db8fa (DEFAULT_TABLE table=default.Table_6500147915384d518c459e558a9ce1d6 [id=98ca5c705ae24203aa9e3946bcbc3bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.688077 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c6f552832a45c4a2e28f40797db8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.691911 5839 ts_tablet_manager.cc:1398] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
W20240507 08:11:56.692870 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 972bc0a5ba924e0fb946cdacd9575c1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 972bc0a5ba924e0fb946cdacd9575c1e already in progress: creating tablet
W20240507 08:11:56.693928 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b2df872d04b94c30872b03367df3cc31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2df872d04b94c30872b03367df3cc31 already in progress: creating tablet
I20240507 08:11:56.698627 5582 raft_consensus.cc:3037] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.699609 5839 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.701332 5683 tablet_service.cc:1467] Processing CreateTablet for tablet b8c6f552832a45c4a2e28f40797db8fa (DEFAULT_TABLE table=default.Table_6500147915384d518c459e558a9ce1d6 [id=98ca5c705ae24203aa9e3946bcbc3bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.701817 5839 tablet_bootstrap.cc:654] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.701855 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c6f552832a45c4a2e28f40797db8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.702469 5410 tablet_service.cc:1467] Processing CreateTablet for tablet b8c6f552832a45c4a2e28f40797db8fa (DEFAULT_TABLE table=default.Table_6500147915384d518c459e558a9ce1d6 [id=98ca5c705ae24203aa9e3946bcbc3bbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.702523 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b (DEFAULT_TABLE table=default.Table_90a87f2fadb249189e0eb022d3730409 [id=64f54c3a3fba416f8810335c6f651ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.702877 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8c6f552832a45c4a2e28f40797db8fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.702877 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5439844f384513b4719cfe4f4e5d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.702934 6226 raft_consensus.cc:482] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.703106 6226 raft_consensus.cc:504] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.704247 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270a1fd15925416483c211021549315c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:56.704605 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "270a1fd15925416483c211021549315c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.704842 5377 leader_election.cc:341] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.706477 6226 leader_election.cc:288] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.706964 6226 raft_consensus.cc:482] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.709411 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b (DEFAULT_TABLE table=default.Table_90a87f2fadb249189e0eb022d3730409 [id=64f54c3a3fba416f8810335c6f651ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.709823 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5439844f384513b4719cfe4f4e5d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.709798 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b (DEFAULT_TABLE table=default.Table_90a87f2fadb249189e0eb022d3730409 [id=64f54c3a3fba416f8810335c6f651ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.710309 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e5439844f384513b4719cfe4f4e5d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.710717 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 99e4514039304f0696c69a057db0ea6e (DEFAULT_TABLE table=default.Table_fc64369fe72742ccb59a630ee95a8007 [id=54ceed7b5cad41ecbacc40ace9f48cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.711091 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e4514039304f0696c69a057db0ea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.713352 6240 consensus_queue.cc:1035] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:11:56.717555 5378 leader_election.cc:341] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.717844 6226 raft_consensus.cc:504] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.718740 5378 leader_election.cc:302] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.722486 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:56.722751 5716 raft_consensus.cc:2454] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:56.723503 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:56.723871 5840 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.723829 5377 leader_election.cc:302] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:56.724577 6240 raft_consensus.cc:2726] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.724812 6240 raft_consensus.cc:2781] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:56.724915 6240 raft_consensus.cc:482] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:56.728389 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 14929efb0ed049d6b554598a5b8cb494 (DEFAULT_TABLE table=default.Table_b21b58e8d49943268e7491117d5d717e [id=afe7e0cf53aa444f84f4773519b48c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.728717 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 14929efb0ed049d6b554598a5b8cb494 (DEFAULT_TABLE table=default.Table_b21b58e8d49943268e7491117d5d717e [id=afe7e0cf53aa444f84f4773519b48c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.729219 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14929efb0ed049d6b554598a5b8cb494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.725029 6240 raft_consensus.cc:3037] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.728005 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 99e4514039304f0696c69a057db0ea6e (DEFAULT_TABLE table=default.Table_fc64369fe72742ccb59a630ee95a8007 [id=54ceed7b5cad41ecbacc40ace9f48cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.730774 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e4514039304f0696c69a057db0ea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.733366 6240 raft_consensus.cc:504] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:56.734748 5378 leader_election.cc:341] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.729713 6226 leader_election.cc:288] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.737401 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 99e4514039304f0696c69a057db0ea6e (DEFAULT_TABLE table=default.Table_fc64369fe72742ccb59a630ee95a8007 [id=54ceed7b5cad41ecbacc40ace9f48cd5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.737857 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99e4514039304f0696c69a057db0ea6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.742391 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14929efb0ed049d6b554598a5b8cb494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.748214 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:11:56.748757 5256 catalog_manager.cc:6554] ReplaceTablet: tablet b7947ca2d8324dbb9c909ec50852fed8 deleted and replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e
I20240507 08:11:56.749121 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:56.749709 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c534708f1abd4a33965eea92bdc0d0de
W20240507 08:11:56.749955 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b7947ca2d8324dbb9c909ec50852fed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7947ca2d8324dbb9c909ec50852fed8 already in progress: creating tablet
I20240507 08:11:56.751070 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet c534708f1abd4a33965eea92bdc0d0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:56.751843 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c534708f1abd4a33965eea92bdc0d0de because the tablet was not found. No further retry: Not found: Tablet not found: c534708f1abd4a33965eea92bdc0d0de
I20240507 08:11:56.751963 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet c534708f1abd4a33965eea92bdc0d0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:56.751858 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:56.752645 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c534708f1abd4a33965eea92bdc0d0de because the tablet was not found. No further retry: Not found: Tablet not found: c534708f1abd4a33965eea92bdc0d0de
I20240507 08:11:56.753159 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:56.753361 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b7947ca2d8324dbb9c909ec50852fed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7947ca2d8324dbb9c909ec50852fed8 already in progress: creating tablet
I20240507 08:11:56.753640 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet c534708f1abd4a33965eea92bdc0d0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:56.754952 5840 tablet_bootstrap.cc:654] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:56.755326 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b7947ca2d8324dbb9c909ec50852fed8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b7947ca2d8324dbb9c909ec50852fed8 already in progress: creating tablet
W20240507 08:11:56.755601 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c534708f1abd4a33965eea92bdc0d0de because the tablet was not found. No further retry: Not found: Tablet not found: c534708f1abd4a33965eea92bdc0d0de
I20240507 08:11:56.757970 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf (DEFAULT_TABLE table=default.Table_a9968c4db5a54a7db162505d22dd31dc [id=d26d851d0f5244419cc308132e26e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.758167 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 242ea35c61404a91b1211c5ad9207728 (DEFAULT_TABLE table=default.Table_5d4ba83817d04005be08381a5c9c15b3 [id=a79c79866e6547a2b0ea94eeaec8c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.758425 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab94b1a09dd4969b928b017229ce4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.758726 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242ea35c61404a91b1211c5ad9207728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.762822 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:56.763392 5378 leader_election.cc:341] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.757205 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 14929efb0ed049d6b554598a5b8cb494 (DEFAULT_TABLE table=default.Table_b21b58e8d49943268e7491117d5d717e [id=afe7e0cf53aa444f84f4773519b48c85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.766896 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14929efb0ed049d6b554598a5b8cb494. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.769294 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf (DEFAULT_TABLE table=default.Table_a9968c4db5a54a7db162505d22dd31dc [id=d26d851d0f5244419cc308132e26e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.769762 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab94b1a09dd4969b928b017229ce4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.770627 6155 consensus_queue.cc:1035] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:56.772099 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf (DEFAULT_TABLE table=default.Table_a9968c4db5a54a7db162505d22dd31dc [id=d26d851d0f5244419cc308132e26e5e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.772576 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab94b1a09dd4969b928b017229ce4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.775024 6155 raft_consensus.cc:482] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:56.776502 6155 raft_consensus.cc:504] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.777643 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf0e2936324a9fbb2426a8026886fc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:56.778227 5378 leader_election.cc:341] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.778648 6155 leader_election.cc:288] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.779433 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7eaf0e2936324a9fbb2426a8026886fc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:56.779965 5377 leader_election.cc:341] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:56.780094 5377 leader_election.cc:302] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.780656 6240 leader_election.cc:288] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:56.782431 6240 raft_consensus.cc:2726] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:56.783411 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:56.783612 5718 raft_consensus.cc:3037] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:56.784372 5547 tablet_service.cc:1467] Processing CreateTablet for tablet c839c8a5041f4c3da9bc0749afee1413 (DEFAULT_TABLE table=default.Table_e19ac67deb044b61b0e6194e8b54aae0 [id=9c2cb8887f5344ff8a6d58b3f7a72459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.784777 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839c8a5041f4c3da9bc0749afee1413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.792340 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7b8c560f048f4b02febd371aa2a8c (DEFAULT_TABLE table=default.Table_5f94711838f74fb88cbb91fe4e7ff11b [id=8d1737f80fdd48d3bf74738c2b5e7f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.792831 5718 raft_consensus.cc:2454] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:56.792851 5696 tablet_service.cc:1467] Processing CreateTablet for tablet c839c8a5041f4c3da9bc0749afee1413 (DEFAULT_TABLE table=default.Table_e19ac67deb044b61b0e6194e8b54aae0 [id=9c2cb8887f5344ff8a6d58b3f7a72459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.793383 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839c8a5041f4c3da9bc0749afee1413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.792867 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 242ea35c61404a91b1211c5ad9207728 (DEFAULT_TABLE table=default.Table_5d4ba83817d04005be08381a5c9c15b3 [id=a79c79866e6547a2b0ea94eeaec8c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.798022 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242ea35c61404a91b1211c5ad9207728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.799456 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7b8c560f048f4b02febd371aa2a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.799494 5377 leader_election.cc:302] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:56.790084 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 242ea35c61404a91b1211c5ad9207728 (DEFAULT_TABLE table=default.Table_5d4ba83817d04005be08381a5c9c15b3 [id=a79c79866e6547a2b0ea94eeaec8c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.792200 5415 tablet_service.cc:1467] Processing CreateTablet for tablet c839c8a5041f4c3da9bc0749afee1413 (DEFAULT_TABLE table=default.Table_e19ac67deb044b61b0e6194e8b54aae0 [id=9c2cb8887f5344ff8a6d58b3f7a72459]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.800179 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 242ea35c61404a91b1211c5ad9207728. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.800457 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c839c8a5041f4c3da9bc0749afee1413. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.803659 6240 raft_consensus.cc:2781] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:56.805984 6240 raft_consensus.cc:686] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:56.806315 6240 consensus_queue.cc:237] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.808410 5839 tablet_bootstrap.cc:492] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.808581 5839 ts_tablet_manager.cc:1367] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.109s user 0.002s sys 0.006s
I20240507 08:11:56.809473 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb (DEFAULT_TABLE table=default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 [id=a286976f28f7468784296bf1ada7bd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.809677 5839 raft_consensus.cc:348] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.809973 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385f4c1eb7f4493a924b813734ddcfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.813918 5256 catalog_manager.cc:5484] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:56.814402 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:56.814702 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_3b339252e2644a438c81a546b3cdb046 [id=0a1bc06a351549d5aea075ad8d026476]). Expected version 1 got 0
I20240507 08:11:56.814862 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:56.819469 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7b8c560f048f4b02febd371aa2a8c (DEFAULT_TABLE table=default.Table_5f94711838f74fb88cbb91fe4e7ff11b [id=8d1737f80fdd48d3bf74738c2b5e7f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.821451 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb (DEFAULT_TABLE table=default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 [id=a286976f28f7468784296bf1ada7bd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.827121 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7b8c560f048f4b02febd371aa2a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.809973 5839 raft_consensus.cc:374] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.828634 5839 raft_consensus.cc:729] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.830472 5839 consensus_queue.cc:260] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:56.835407 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8ebbcf0d5a54d63a96a9374e839af83 consensus state has no leader
I20240507 08:11:56.837765 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_59e07230b27a4e67acb956550768595d" } new_table_name: "default.Table_54adb6f25ed342928404f4a1853c9a60" modify_external_catalogs: true
I20240507 08:11:56.838927 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385f4c1eb7f4493a924b813734ddcfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.837543 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_499cbcb48a784dcab2336eece710d4b1" } modify_external_catalogs: true
I20240507 08:11:56.839270 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_499cbcb48a784dcab2336eece710d4b1" } modify_external_catalogs: true
I20240507 08:11:56.841440 5839 ts_tablet_manager.cc:1398] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
W20240507 08:11:56.841835 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b60ec15a83934999871425ef9a64ec72 has no consensus state
W20240507 08:11:56.842377 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 916e33e40ef44d06a359b82d8fe48157 consensus state has no leader
W20240507 08:11:56.844244 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1c6b488b9014841899597d858bda5e2 consensus state has no leader
I20240507 08:11:56.845765 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_361a1e87a0ae409dbfa844e2b985885e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.846307 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_361a1e87a0ae409dbfa844e2b985885e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.847774 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d74868790e444b1fb50ea56051f4810c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.848214 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d74868790e444b1fb50ea56051f4810c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.848315 5839 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.849705 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6f159b4cb58147b884dd65d03fe08c54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.850154 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f159b4cb58147b884dd65d03fe08c54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.853402 5839 tablet_bootstrap.cc:654] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.854593 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_825e44ca78c14d6392c7e6864babde4c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:56.855068 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_825e44ca78c14d6392c7e6864babde4c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:56.861488 5840 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.861706 5840 ts_tablet_manager.cc:1367] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.002s
I20240507 08:11:56.862493 5840 raft_consensus.cc:348] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.862707 5840 raft_consensus.cc:374] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.862807 5840 raft_consensus.cc:729] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.863111 5840 consensus_queue.cc:260] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.864157 5840 ts_tablet_manager.cc:1398] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:56.864957 5840 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.866722 5840 tablet_bootstrap.cc:654] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.868435 5559 tablet_service.cc:1467] Processing CreateTablet for tablet b0542325c80241fca7e317cda64302a9 (DEFAULT_TABLE table=default.Table_8c28748ca00b43e7bff97d436034daaf [id=657d76ac3c524ee6988bda523bbd19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.868896 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0542325c80241fca7e317cda64302a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.872303 5839 tablet_bootstrap.cc:492] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.872542 5839 ts_tablet_manager.cc:1367] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240507 08:11:56.872723 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb (DEFAULT_TABLE table=default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 [id=a286976f28f7468784296bf1ada7bd29]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.873160 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 385f4c1eb7f4493a924b813734ddcfdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.873601 5839 raft_consensus.cc:348] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.873890 5839 raft_consensus.cc:374] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.874028 5839 raft_consensus.cc:729] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.875456 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (DEFAULT_TABLE table=default.Table_75025136df1f453e92ffc67c475fbb56 [id=f274fff64ffc4f0eac28cbb1115fc0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.875941 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6454c4a59d4c4cae1681bcae0da2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.874313 5839 consensus_queue.cc:260] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.877071 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 2a77151b173d448aa247317ec6160ce0 (DEFAULT_TABLE table=default.Table_63157d8c47004724ae6c531116bbf7dd [id=2c8959dfe4d34791beb0d59b8f52c3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.877447 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a77151b173d448aa247317ec6160ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.878209 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 7db7b8c560f048f4b02febd371aa2a8c (DEFAULT_TABLE table=default.Table_5f94711838f74fb88cbb91fe4e7ff11b [id=8d1737f80fdd48d3bf74738c2b5e7f13]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.878587 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7db7b8c560f048f4b02febd371aa2a8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.878891 5839 ts_tablet_manager.cc:1398] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240507 08:11:56.879730 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2a77151b173d448aa247317ec6160ce0 (DEFAULT_TABLE table=default.Table_63157d8c47004724ae6c531116bbf7dd [id=2c8959dfe4d34791beb0d59b8f52c3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.880144 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a77151b173d448aa247317ec6160ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.881261 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (DEFAULT_TABLE table=default.Table_75025136df1f453e92ffc67c475fbb56 [id=f274fff64ffc4f0eac28cbb1115fc0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.881610 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6454c4a59d4c4cae1681bcae0da2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.881909 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (DEFAULT_TABLE table=default.Table_75025136df1f453e92ffc67c475fbb56 [id=f274fff64ffc4f0eac28cbb1115fc0a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.885756 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e6454c4a59d4c4cae1681bcae0da2d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.888199 5683 tablet_service.cc:1467] Processing CreateTablet for tablet b0542325c80241fca7e317cda64302a9 (DEFAULT_TABLE table=default.Table_8c28748ca00b43e7bff97d436034daaf [id=657d76ac3c524ee6988bda523bbd19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.888549 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0542325c80241fca7e317cda64302a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.901413 5689 tablet_service.cc:1467] Processing CreateTablet for tablet c534708f1abd4a33965eea92bdc0d0de (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.901893 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534708f1abd4a33965eea92bdc0d0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.903898 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 2a77151b173d448aa247317ec6160ce0 (DEFAULT_TABLE table=default.Table_63157d8c47004724ae6c531116bbf7dd [id=2c8959dfe4d34791beb0d59b8f52c3de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.904349 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a77151b173d448aa247317ec6160ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.908159 5696 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4eb72b855459b8da46c037108097e (DEFAULT_TABLE table=default.Table_871e9a698bf647929a1db0c34970c457 [id=5bef63e11c734867ab4c67ca05d8e2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.908550 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4eb72b855459b8da46c037108097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.908529 5424 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4eb72b855459b8da46c037108097e (DEFAULT_TABLE table=default.Table_871e9a698bf647929a1db0c34970c457 [id=5bef63e11c734867ab4c67ca05d8e2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.908875 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4eb72b855459b8da46c037108097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.913782 5839 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.915820 5425 tablet_service.cc:1467] Processing CreateTablet for tablet c534708f1abd4a33965eea92bdc0d0de (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.915900 5839 tablet_bootstrap.cc:654] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.916262 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534708f1abd4a33965eea92bdc0d0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.917805 5547 tablet_service.cc:1467] Processing CreateTablet for tablet c534708f1abd4a33965eea92bdc0d0de (DEFAULT_TABLE table=default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.918236 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c534708f1abd4a33965eea92bdc0d0de. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.921880 5559 tablet_service.cc:1467] Processing CreateTablet for tablet b7b4eb72b855459b8da46c037108097e (DEFAULT_TABLE table=default.Table_871e9a698bf647929a1db0c34970c457 [id=5bef63e11c734867ab4c67ca05d8e2e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.922338 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7b4eb72b855459b8da46c037108097e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.922715 5843 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.922894 5843 ts_tablet_manager.cc:1367] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.410s user 0.112s sys 0.120s
I20240507 08:11:56.923943 5843 raft_consensus.cc:348] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.924244 5843 raft_consensus.cc:374] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.924369 5843 raft_consensus.cc:729] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.924659 5843 consensus_queue.cc:260] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.925590 5843 ts_tablet_manager.cc:1398] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240507 08:11:56.925936 5843 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.928540 5843 tablet_bootstrap.cc:654] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.928592 5413 tablet_service.cc:1467] Processing CreateTablet for tablet b0542325c80241fca7e317cda64302a9 (DEFAULT_TABLE table=default.Table_8c28748ca00b43e7bff97d436034daaf [id=657d76ac3c524ee6988bda523bbd19c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.929072 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0542325c80241fca7e317cda64302a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.932008 5843 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.932229 5843 ts_tablet_manager.cc:1367] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20240507 08:11:56.932946 5843 raft_consensus.cc:348] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.933185 5843 raft_consensus.cc:374] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.933265 5843 raft_consensus.cc:729] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.933537 5843 consensus_queue.cc:260] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.934240 5843 ts_tablet_manager.cc:1398] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:56.934547 5843 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.936671 5843 tablet_bootstrap.cc:654] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.937224 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 167ee9cbc9174b82824c468e098472a6 (DEFAULT_TABLE table=default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.939160 5421 tablet_service.cc:1467] Processing CreateTablet for tablet fed715ba993a4f0c9e54b209ef031981 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.939558 5549 tablet_service.cc:1467] Processing CreateTablet for tablet fed715ba993a4f0c9e54b209ef031981 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.939836 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167ee9cbc9174b82824c468e098472a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.940137 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed715ba993a4f0c9e54b209ef031981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.940160 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed715ba993a4f0c9e54b209ef031981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.941982 5561 tablet_service.cc:1467] Processing CreateTablet for tablet accf88112b0b408e8d24828fc93055ef (DEFAULT_TABLE table=default.Table_d0b3de590398468abb0a9e5277b608c9 [id=cf35bf92a1e04866a744a6a678948844]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.942382 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf88112b0b408e8d24828fc93055ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.943217 5689 tablet_service.cc:1467] Processing CreateTablet for tablet accf88112b0b408e8d24828fc93055ef (DEFAULT_TABLE table=default.Table_d0b3de590398468abb0a9e5277b608c9 [id=cf35bf92a1e04866a744a6a678948844]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.943395 5843 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:56.943535 5843 ts_tablet_manager.cc:1367] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.001s
I20240507 08:11:56.943625 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf88112b0b408e8d24828fc93055ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.944600 5843 raft_consensus.cc:348] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.944834 5843 raft_consensus.cc:374] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.944917 5843 raft_consensus.cc:729] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.945133 5843 consensus_queue.cc:260] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.945848 5843 ts_tablet_manager.cc:1398] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:56.945874 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 167ee9cbc9174b82824c468e098472a6 (DEFAULT_TABLE table=default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.946159 5843 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:56.946221 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167ee9cbc9174b82824c468e098472a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.947774 5843 tablet_bootstrap.cc:654] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.947803 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 167ee9cbc9174b82824c468e098472a6 (DEFAULT_TABLE table=default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.948256 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 167ee9cbc9174b82824c468e098472a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.950143 5424 tablet_service.cc:1467] Processing CreateTablet for tablet accf88112b0b408e8d24828fc93055ef (DEFAULT_TABLE table=default.Table_d0b3de590398468abb0a9e5277b608c9 [id=cf35bf92a1e04866a744a6a678948844]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.950747 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet accf88112b0b408e8d24828fc93055ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.956429 5840 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:56.956593 5840 ts_tablet_manager.cc:1367] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.092s user 0.003s sys 0.000s
I20240507 08:11:56.957437 5840 raft_consensus.cc:348] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.957686 5840 raft_consensus.cc:374] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.957839 5840 raft_consensus.cc:729] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.958117 5840 consensus_queue.cc:260] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:56.960716 5840 ts_tablet_manager.cc:1398] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240507 08:11:56.961356 5840 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:56.969424 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb648fe98e74248b5cd65603de58a02 (DEFAULT_TABLE table=default.Table_4737c51589da4300bf37cc2638784041 [id=58501826cb5744918aba9bf338a0eccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.969625 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb648fe98e74248b5cd65603de58a02 (DEFAULT_TABLE table=default.Table_4737c51589da4300bf37cc2638784041 [id=58501826cb5744918aba9bf338a0eccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.969892 5839 tablet_bootstrap.cc:492] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:56.970023 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb648fe98e74248b5cd65603de58a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.970113 5839 ts_tablet_manager.cc:1367] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.002s
I20240507 08:11:56.970216 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb648fe98e74248b5cd65603de58a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.976608 5839 raft_consensus.cc:348] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.976922 5839 raft_consensus.cc:374] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:56.977052 5839 raft_consensus.cc:729] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:56.976861 5699 tablet_service.cc:1467] Processing CreateTablet for tablet fed715ba993a4f0c9e54b209ef031981 (DEFAULT_TABLE table=default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.977521 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fed715ba993a4f0c9e54b209ef031981. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.978204 5839 consensus_queue.cc:260] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:56.983201 5840 tablet_bootstrap.cc:654] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:56.986570 5839 ts_tablet_manager.cc:1398] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20240507 08:11:56.993878 5839 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:56.994836 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd3dbd84234e398a6151483d447260 (DEFAULT_TABLE table=default.Table_6b4f73d517f14c499174dbe4f4ede6c3 [id=fbfc49165406411691760a333e8a098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.995240 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd3dbd84234e398a6151483d447260 (DEFAULT_TABLE table=default.Table_6b4f73d517f14c499174dbe4f4ede6c3 [id=fbfc49165406411691760a333e8a098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:56.995671 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd3dbd84234e398a6151483d447260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.995677 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd3dbd84234e398a6151483d447260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:56.995901 5839 tablet_bootstrap.cc:654] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.002259 5839 tablet_bootstrap.cc:492] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.002508 5839 ts_tablet_manager.cc:1367] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20240507 08:11:57.003546 5839 raft_consensus.cc:348] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.004017 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:57.004030 5839 raft_consensus.cc:374] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.004165 5839 raft_consensus.cc:729] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.004984 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 97c978ec8bca4aa5b2ffaf22c3290747 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:57.005940 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 47a154769d0f4501837888feba98f33d (DEFAULT_TABLE table=default.Table_3a7027841ed1443096999162f8cdd1ea [id=567b19917c9148d4ad6ff350294bf568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.006443 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_f69f572cb6f6424eb22ea998d9f6ba3f [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 1 got 0
I20240507 08:11:57.007047 5839 consensus_queue.cc:260] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.010324 5698 tablet_service.cc:1467] Processing CreateTablet for tablet aca8c38841db45e38ae5d8d965a16d0e (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.010763 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 7d89e4185229431f9a68f599da4aff34 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.012111 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a154769d0f4501837888feba98f33d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.013993 5552 tablet_service.cc:1467] Processing CreateTablet for tablet aca8c38841db45e38ae5d8d965a16d0e (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.014583 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca8c38841db45e38ae5d8d965a16d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.014772 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca8c38841db45e38ae5d8d965a16d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.015630 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d89e4185229431f9a68f599da4aff34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.016539 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_336e8bfc36dd4d50a3a52a778ef1f102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.017063 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_336e8bfc36dd4d50a3a52a778ef1f102 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.018100 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cd8f372c67b74794a5ac56adc1990bad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.018596 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd8f372c67b74794a5ac56adc1990bad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.018924 6073 raft_consensus.cc:482] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.019060 6073 raft_consensus.cc:504] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.019912 6073 leader_election.cc:288] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:11:57.021337 5517 leader_election.cc:341] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:57.025681 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet e49dfd218a24464fbac39a3f4b477462 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.025974 5838 tablet_replica.cc:330] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.026182 5838 raft_consensus.cc:2229] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.026382 5838 raft_consensus.cc:2258] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.027119 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 697611671ef64c53b0be4f5341b8f270 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.027659 5838 ts_tablet_manager.cc:1875] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.027884 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 80d82625304942d38b0261f95e235a41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.030093 5256 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_0856317e9911474aa5861773d7185325 [id=0a1bc06a351549d5aea075ad8d026476]). Expected version 2 got 0
I20240507 08:11:57.031190 5839 ts_tablet_manager.cc:1398] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20240507 08:11:57.027655 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 7d89e4185229431f9a68f599da4aff34 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.032233 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 47a154769d0f4501837888feba98f33d (DEFAULT_TABLE table=default.Table_3a7027841ed1443096999162f8cdd1ea [id=567b19917c9148d4ad6ff350294bf568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.032918 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4fdb46838aab420bb1038ea941103c52" } modify_external_catalogs: true
I20240507 08:11:57.033113 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4fdb46838aab420bb1038ea941103c52" } modify_external_catalogs: true
I20240507 08:11:57.034570 5838 ts_tablet_manager.cc:1888] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.034677 5838 log.cc:1196] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:57.035434 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ba24c078af9472da06793c58f147290 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:57.035777 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03a59c7257e74b4abaa4ea7a1ee9136d consensus state has no leader
I20240507 08:11:57.036233 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_307c9baefe434bb68c8d0906a9b9788e" } modify_external_catalogs: true
I20240507 08:11:57.036419 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_307c9baefe434bb68c8d0906a9b9788e" } modify_external_catalogs: true
I20240507 08:11:57.036775 5838 ts_tablet_manager.cc:1909] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.037047 5839 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.038110 5838 tablet_replica.cc:330] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.038331 5838 raft_consensus.cc:2229] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.038401 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e49dfd218a24464fbac39a3f4b477462 (table default.Table_f053f92b620148968b86e702410b211d [id=d5c4c2713c2448c5805ed46efed5a4cd]) successfully deleted
I20240507 08:11:57.038529 5838 raft_consensus.cc:2258] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.040618 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8a015bf84a454d49b6497c16ed15c84a
I20240507 08:11:57.040967 5838 ts_tablet_manager.cc:1875] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.042708 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e19ac67deb044b61b0e6194e8b54aae0" } modify_external_catalogs: true
I20240507 08:11:57.042876 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e19ac67deb044b61b0e6194e8b54aae0" } modify_external_catalogs: true
I20240507 08:11:57.043244 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.045173 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.045442 5837 tablet_replica.cc:330] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.045426 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.045665 5837 raft_consensus.cc:2229] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:57.045693 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8a015bf84a454d49b6497c16ed15c84a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a015bf84a454d49b6497c16ed15c84a already in progress: creating tablet
I20240507 08:11:57.045876 5837 raft_consensus.cc:2258] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.046329 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet f29568ebe4774662a8b49b7829cf30e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.046890 5837 ts_tablet_manager.cc:1875] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:57.056917 5520 leader_election.cc:341] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e49dfd218a24464fbac39a3f4b477462
I20240507 08:11:57.057122 5520 leader_election.cc:302] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:11:57.057348 5520 raft_consensus.cc:2670] T e49dfd218a24464fbac39a3f4b477462 P fe673739d8084e7da5d7edd722e857d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:57.059844 5837 ts_tablet_manager.cc:1888] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.060014 5837 log.cc:1196] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:57.060369 5837 ts_tablet_manager.cc:1909] T 4867e1cca1634b4ab6e36f9ed3204e0a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.061442 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4867e1cca1634b4ab6e36f9ed3204e0a (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
I20240507 08:11:57.062146 5840 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.062297 5840 ts_tablet_manager.cc:1367] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.000s
I20240507 08:11:57.062944 5840 raft_consensus.cc:348] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.063150 5840 raft_consensus.cc:374] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.063073 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 47a154769d0f4501837888feba98f33d (DEFAULT_TABLE table=default.Table_3a7027841ed1443096999162f8cdd1ea [id=567b19917c9148d4ad6ff350294bf568]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.063211 5840 raft_consensus.cc:729] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.063166 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 2bb648fe98e74248b5cd65603de58a02 (DEFAULT_TABLE table=default.Table_4737c51589da4300bf37cc2638784041 [id=58501826cb5744918aba9bf338a0eccf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.063549 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a154769d0f4501837888feba98f33d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.063462 5840 consensus_queue.cc:260] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.064076 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bb648fe98e74248b5cd65603de58a02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.065666 5840 ts_tablet_manager.cc:1398] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:57.066334 5840 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:57.069274 5840 tablet_bootstrap.cc:654] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.070784 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.071203 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 4e63c3f3ecc84913ad05864be450e9b0 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:57.071656 5841 tablet_replica.cc:330] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:57.071915 5841 raft_consensus.cc:2229] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.074182 5841 raft_consensus.cc:2258] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.074651 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet bc619db3a2944ecc9f0112b28de148a6 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:57.074755 5841 ts_tablet_manager.cc:1875] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.074908 5839 tablet_bootstrap.cc:654] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.075994 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:57.075958 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 7d89e4185229431f9a68f599da4aff34 (DEFAULT_TABLE table=default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.076536 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.077232 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.077847 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.078037 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.078352 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d89e4185229431f9a68f599da4aff34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.081009 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.081745 5837 tablet_replica.cc:330] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.081969 5837 raft_consensus.cc:2229] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.082183 5837 raft_consensus.cc:2258] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.082636 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47a154769d0f4501837888feba98f33d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.086120 5837 ts_tablet_manager.cc:1875] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.089459 5841 ts_tablet_manager.cc:1888] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.089815 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d89e4185229431f9a68f599da4aff34. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.090387 5839 tablet_bootstrap.cc:492] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.090557 5839 ts_tablet_manager.cc:1367] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.010s
I20240507 08:11:57.091611 5839 raft_consensus.cc:348] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.091818 5839 raft_consensus.cc:374] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.091915 5839 raft_consensus.cc:729] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.092140 5839 consensus_queue.cc:260] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.092805 5839 ts_tablet_manager.cc:1398] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:57.093375 5839 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.099761 5843 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.099944 5843 ts_tablet_manager.cc:1367] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.154s user 0.032s sys 0.066s
I20240507 08:11:57.100715 5843 raft_consensus.cc:348] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.100944 5843 raft_consensus.cc:374] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.101018 5843 raft_consensus.cc:729] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.101296 5843 consensus_queue.cc:260] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.102950 5838 ts_tablet_manager.cc:1888] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.103087 5838 log.cc:1196] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/697611671ef64c53b0be4f5341b8f270
I20240507 08:11:57.103533 5838 ts_tablet_manager.cc:1909] T 697611671ef64c53b0be4f5341b8f270 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.104203 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 697611671ef64c53b0be4f5341b8f270 (table default.Table_c3dd460f347c47e9990f8cd2f4aeae71 [id=8ead2c62edac4d969405609231e13dfc]) successfully deleted
I20240507 08:11:57.104409 5843 ts_tablet_manager.cc:1398] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:11:57.104717 5843 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.109375 5839 tablet_bootstrap.cc:654] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.113021 5840 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.113186 5840 ts_tablet_manager.cc:1367] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240507 08:11:57.114045 5840 raft_consensus.cc:348] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.114290 5840 raft_consensus.cc:374] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.114382 5840 raft_consensus.cc:729] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.114624 5840 consensus_queue.cc:260] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.117534 5840 ts_tablet_manager.cc:1398] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:57.118762 5840 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:57.122174 5838 tablet_replica.cc:330] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.122406 5838 raft_consensus.cc:2229] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.122613 5838 raft_consensus.cc:2258] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.122823 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 97c978ec8bca4aa5b2ffaf22c3290747
I20240507 08:11:57.123368 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 97c978ec8bca4aa5b2ffaf22c3290747 deleted and replaced by tablet aa806508ec3e489a9b2c2fa9196198d5
I20240507 08:11:57.124851 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.126185 5838 ts_tablet_manager.cc:1875] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.127339 5843 tablet_bootstrap.cc:654] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.121618 5837 ts_tablet_manager.cc:1888] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.126120 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.130964 5837 log.cc:1196] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f922a7519084409db98693ee8797592b
I20240507 08:11:57.131378 5837 ts_tablet_manager.cc:1909] T f922a7519084409db98693ee8797592b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.132572 5839 tablet_bootstrap.cc:492] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.132661 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f922a7519084409db98693ee8797592b (table default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]) successfully deleted
I20240507 08:11:57.132839 5839 ts_tablet_manager.cc:1367] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.000s
W20240507 08:11:57.133536 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 97c978ec8bca4aa5b2ffaf22c3290747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c978ec8bca4aa5b2ffaf22c3290747 already in progress: creating tablet
I20240507 08:11:57.133785 5839 raft_consensus.cc:348] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.134055 5839 raft_consensus.cc:374] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.134199 5839 raft_consensus.cc:729] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.134505 5839 consensus_queue.cc:260] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.136329 5839 ts_tablet_manager.cc:1398] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:57.136674 5839 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.137607 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.138489 5839 tablet_bootstrap.cc:654] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.143440 5840 tablet_bootstrap.cc:654] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.145656 5422 tablet_service.cc:1467] Processing CreateTablet for tablet aca8c38841db45e38ae5d8d965a16d0e (DEFAULT_TABLE table=default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.146238 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aca8c38841db45e38ae5d8d965a16d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.150210 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b1fadbfd330b4e699e151f83125da281
I20240507 08:11:57.150717 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c839c8a5041f4c3da9bc0749afee1413
I20240507 08:11:57.151791 5261 catalog_manager.cc:5484] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:57.152406 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_086d5c28a17243c9991e0add9910274a [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 2 got 0
I20240507 08:11:57.152909 5261 catalog_manager.cc:5484] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:57.153411 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:57.154630 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2da5736a4b6944c29d94f8b89cc4e638" } modify_external_catalogs: true
I20240507 08:11:57.154832 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2da5736a4b6944c29d94f8b89cc4e638" } modify_external_catalogs: true
I20240507 08:11:57.157821 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.158223 5256 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet e964cccfc46c46819384ce0de403eaf2
I20240507 08:11:57.160322 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ca8fc2c7f3849c380fea56862ffc26e with delete_type TABLET_DATA_DELETED (replaced by tablet 167ee9cbc9174b82824c468e098472a6) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.160851 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.161672 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.162508 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.162932 5841 log.cc:1196] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f922a7519084409db98693ee8797592b
I20240507 08:11:57.162927 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.163151 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.163496 5841 ts_tablet_manager.cc:1909] T f922a7519084409db98693ee8797592b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:57.164991 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f922a7519084409db98693ee8797592b (table default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]) successfully deleted
W20240507 08:11:57.165297 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f922a7519084409db98693ee8797592b because the tablet was not found. No further retry: Not found: Tablet not found: f922a7519084409db98693ee8797592b
I20240507 08:11:57.165541 5841 tablet_replica.cc:330] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:57.165916 5841 raft_consensus.cc:2229] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:11:57.165935 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b1fadbfd330b4e699e151f83125da281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fadbfd330b4e699e151f83125da281 already in progress: creating tablet
I20240507 08:11:57.166173 5841 raft_consensus.cc:2258] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:57.166425 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c839c8a5041f4c3da9bc0749afee1413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c839c8a5041f4c3da9bc0749afee1413 already in progress: creating tablet
I20240507 08:11:57.166898 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.166988 5841 ts_tablet_manager.cc:1875] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.169014 5838 ts_tablet_manager.cc:1888] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.169909 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4fb1a5ebaf7c4b01bd7123a48782485f" } new_table_name: "default.Table_291925856aa04048bb7d289bb6837ec0" modify_external_catalogs: true
I20240507 08:11:57.169865 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 66cd3dbd84234e398a6151483d447260 (DEFAULT_TABLE table=default.Table_6b4f73d517f14c499174dbe4f4ede6c3 [id=fbfc49165406411691760a333e8a098c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.170255 5838 log.cc:1196] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/80d82625304942d38b0261f95e235a41
I20240507 08:11:57.170480 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66cd3dbd84234e398a6151483d447260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.170632 5838 ts_tablet_manager.cc:1909] T 80d82625304942d38b0261f95e235a41 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.171285 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c1bde6a097f946a1bae0b57150f3aef5" } new_table_name: "default.Table_5b58be7978724a40b7c1020659a68b1c" modify_external_catalogs: true
I20240507 08:11:57.173857 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.174360 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.174597 5995 raft_consensus.cc:482] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.174784 5995 raft_consensus.cc:504] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.175626 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_101fc94f9f8b4a70886064fc2f62b7de" } new_table_name: "default.Table_a5fe82b3814f4ba4be0ad9369bbff3e8" modify_external_catalogs: true
I20240507 08:11:57.175626 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4010eb808d9c46979560be92d9aa7613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.176152 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4010eb808d9c46979560be92d9aa7613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.176715 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49dd4772f8e46478638b29c009c83ea" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:57.176844 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 80d82625304942d38b0261f95e235a41 (table default.Table_014d10f31cfe4b05a42d8f9c3d3921e3 [id=478717ae383042c8848a113d90cc85ed]) successfully deleted
W20240507 08:11:57.177589 5377 leader_election.cc:341] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.178056 5995 leader_election.cc:288] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.175927 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f49dd4772f8e46478638b29c009c83ea" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:57.179638 5378 leader_election.cc:341] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.179775 5378 leader_election.cc:302] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:57.180264 5995 raft_consensus.cc:2726] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:57.175381 5838 tablet_replica.cc:330] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.180729 5838 raft_consensus.cc:2229] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.180970 5838 raft_consensus.cc:2258] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.182521 5839 tablet_bootstrap.cc:492] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.182710 5839 ts_tablet_manager.cc:1367] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.046s user 0.013s sys 0.003s
I20240507 08:11:57.183519 5839 raft_consensus.cc:348] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.183732 5839 raft_consensus.cc:374] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.183840 5839 raft_consensus.cc:729] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:11:57.185134 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240507 08:11:57.186796 5839 consensus_queue.cc:260] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.188231 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4e63c3f3ecc84913ad05864be450e9b0
I20240507 08:11:57.188330 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bc619db3a2944ecc9f0112b28de148a6
I20240507 08:11:57.188822 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 4e63c3f3ecc84913ad05864be450e9b0 deleted and replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff
I20240507 08:11:57.188931 5262 catalog_manager.cc:6554] ReplaceTablet: tablet bc619db3a2944ecc9f0112b28de148a6 deleted and replaced by tablet 817542fbc9624497b604b0bc2e39340a
I20240507 08:11:57.190678 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.190922 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet bc619db3a2944ecc9f0112b28de148a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 817542fbc9624497b604b0bc2e39340a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.191073 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.191522 5839 ts_tablet_manager.cc:1398] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:11:57.191895 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet bc619db3a2944ecc9f0112b28de148a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 817542fbc9624497b604b0bc2e39340a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.191905 5839 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.193706 5839 tablet_bootstrap.cc:654] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.194198 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aa4603034e7845b493166a810805b43c" } new_table_name: "default.Table_3b2b1ae34b854b27b9a0ab00c78c17bb" modify_external_catalogs: true
I20240507 08:11:57.195889 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet cb7acbdb357445adb9ce69103c782a7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.197548 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6a2e33773a684a20903120d0e1c2e4c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.198089 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a2e33773a684a20903120d0e1c2e4c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.199684 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.199990 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet bc619db3a2944ecc9f0112b28de148a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 817542fbc9624497b604b0bc2e39340a) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:57.202065 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4e63c3f3ecc84913ad05864be450e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e63c3f3ecc84913ad05864be450e9b0 already in progress: creating tablet
I20240507 08:11:57.202316 5837 tablet_replica.cc:330] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.202605 5837 raft_consensus.cc:2229] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.202891 5837 raft_consensus.cc:2258] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.203517 5837 ts_tablet_manager.cc:1875] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.207190 5843 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.207335 5843 ts_tablet_manager.cc:1367] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.103s user 0.002s sys 0.015s
I20240507 08:11:57.208061 5843 raft_consensus.cc:348] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.208241 5843 raft_consensus.cc:374] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.208326 5843 raft_consensus.cc:729] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.208536 5843 consensus_queue.cc:260] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.209025 5843 ts_tablet_manager.cc:1398] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:11:57.209329 5843 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.209759 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.211496 5840 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.211663 5840 ts_tablet_manager.cc:1367] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.000s
I20240507 08:11:57.215294 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:11:57.215816 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 deleted and replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614
I20240507 08:11:57.216620 5263 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet c831cd95f03e493583eea0fad6ae83bc
I20240507 08:11:57.216863 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 4fcc1d7e309946d783a524b61d5f1269 from {username='slave'} at 127.0.0.1:44738
W20240507 08:11:57.217273 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28e5a380f2784e3a806afcbb5cd2519a consensus state has no leader
W20240507 08:11:57.219280 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c4cd7b56d4e468eb754d9653117d023 consensus state has no leader
W20240507 08:11:57.219642 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 has no consensus state
I20240507 08:11:57.222633 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.217463 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_aa9ec40b8ef74e34994df20db4800454"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.223450 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa9ec40b8ef74e34994df20db4800454 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.223560 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet ea17889619fa420fbeacf1b41594c598 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.224071 5837 ts_tablet_manager.cc:1888] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:57.224208 5837 log.cc:1196] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/bc619db3a2944ecc9f0112b28de148a6
I20240507 08:11:57.224581 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.224761 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.224841 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.225298 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.225497 5843 tablet_bootstrap.cc:654] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.226240 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.228437 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.228739 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.229800 5837 ts_tablet_manager.cc:1909] T bc619db3a2944ecc9f0112b28de148a6 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.230346 5838 ts_tablet_manager.cc:1875] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.230861 5837 tablet_replica.cc:330] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.230954 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet bc619db3a2944ecc9f0112b28de148a6 (table default.Table_cb5f53bbcf3049939314d98e891fe83c [id=108648d0bbfe41cd961bdd8b7736989d]) successfully deleted
I20240507 08:11:57.231097 5837 raft_consensus.cc:2229] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.231271 5837 raft_consensus.cc:2258] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.232038 5837 ts_tablet_manager.cc:1875] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.212561 5840 raft_consensus.cc:348] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.232890 5840 raft_consensus.cc:374] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.232985 5840 raft_consensus.cc:729] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.233779 5840 consensus_queue.cc:260] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:11:57.238236 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35e34b60b78b402faf51e003c97db787 consensus state has no leader
I20240507 08:11:57.238619 5841 ts_tablet_manager.cc:1888] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.238811 5841 log.cc:1196] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:11:57.239629 5841 ts_tablet_manager.cc:1909] T 7eaf0e2936324a9fbb2426a8026886fc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:57.241875 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1ccda05e59e0486b8a26054c5514666f" } new_table_name: "default.Table_166bf386f4e046d7a019c2b3a2689225" modify_external_catalogs: true
I20240507 08:11:57.244349 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.246605 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c799c3ed21d4c66bef27973c9e4688d" } modify_external_catalogs: true
I20240507 08:11:57.246907 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c799c3ed21d4c66bef27973c9e4688d" } modify_external_catalogs: true
I20240507 08:11:57.247555 5840 ts_tablet_manager.cc:1398] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20240507 08:11:57.247932 5840 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:57.240934 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7eaf0e2936324a9fbb2426a8026886fc (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:11:57.253203 5840 tablet_bootstrap.cc:654] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:57.253743 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 835d7b779d724df4863bb13a5140dbf0 consensus state has no leader
W20240507 08:11:57.256166 5517 consensus_peers.cc:485] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bc619db3a2944ecc9f0112b28de148a6. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:57.257541 5837 ts_tablet_manager.cc:1888] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.258531 5837 log.cc:1196] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d9ff5e1b6a9a476c8295a0d0ee85aecc
I20240507 08:11:57.259100 5837 ts_tablet_manager.cc:1909] T d9ff5e1b6a9a476c8295a0d0ee85aecc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.260489 5843 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.260668 5843 ts_tablet_manager.cc:1367] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.007s
I20240507 08:11:57.261484 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (table default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]) successfully deleted
I20240507 08:11:57.262004 5843 raft_consensus.cc:348] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:57.262113 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 already in progress: creating tablet
I20240507 08:11:57.262231 5843 raft_consensus.cc:374] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.262315 5843 raft_consensus.cc:729] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.262493 5837 tablet_replica.cc:330] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.262591 5843 consensus_queue.cc:260] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.262848 5841 tablet_replica.cc:330] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:57.263214 5843 ts_tablet_manager.cc:1398] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:57.263449 5841 raft_consensus.cc:2229] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.263818 5837 raft_consensus.cc:2229] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.263648 5841 raft_consensus.cc:2258] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.264005 5837 raft_consensus.cc:2258] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.264261 5838 ts_tablet_manager.cc:1888] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.264415 5841 ts_tablet_manager.cc:1875] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.264667 5838 log.cc:1196] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/9ba24c078af9472da06793c58f147290
I20240507 08:11:57.265085 5838 ts_tablet_manager.cc:1909] T 9ba24c078af9472da06793c58f147290 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.265512 5843 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.267138 5839 tablet_bootstrap.cc:492] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.267292 5837 ts_tablet_manager.cc:1875] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.267478 5839 ts_tablet_manager.cc:1367] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.009s
I20240507 08:11:57.268101 5839 raft_consensus.cc:348] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.268350 5839 raft_consensus.cc:374] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.268426 5839 raft_consensus.cc:729] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.268693 5839 consensus_queue.cc:260] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.270541 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7c7240e1d89c4c05a124c7126954e1ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.271090 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c7240e1d89c4c05a124c7126954e1ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.273911 5839 ts_tablet_manager.cc:1398] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240507 08:11:57.274518 5840 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.274703 5840 ts_tablet_manager.cc:1367] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240507 08:11:57.276181 5840 raft_consensus.cc:348] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.276387 5840 raft_consensus.cc:374] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.276495 5840 raft_consensus.cc:729] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.277082 5840 consensus_queue.cc:260] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.277871 5839 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.279464 5839 tablet_bootstrap.cc:654] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.281147 5838 tablet_replica.cc:330] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.281416 5838 raft_consensus.cc:2229] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.281705 5838 raft_consensus.cc:2258] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.283562 5837 ts_tablet_manager.cc:1888] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.283686 5837 log.cc:1196] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8e3ffaf1781e4b108f305346476905be
I20240507 08:11:57.283535 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.284085 5837 ts_tablet_manager.cc:1909] T 8e3ffaf1781e4b108f305346476905be P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.284101 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_479f84438e6548f1806133a52949d796"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.282593 5838 ts_tablet_manager.cc:1875] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:57.284729 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_479f84438e6548f1806133a52949d796 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.285600 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 9ba24c078af9472da06793c58f147290 (table default.Table_70320eb4c56d4c38b5f30235746e7cb2 [id=8ea36271c6814ba58435556d3ff59f2a]) successfully deleted
W20240507 08:11:57.285989 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8a015bf84a454d49b6497c16ed15c84a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a015bf84a454d49b6497c16ed15c84a already in progress: creating tablet
W20240507 08:11:57.284268 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.288225 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.288513 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8e3ffaf1781e4b108f305346476905be (table default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]) successfully deleted
W20240507 08:11:57.288781 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 270a1fd15925416483c211021549315c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 270a1fd15925416483c211021549315c already in progress: creating tablet
I20240507 08:11:57.293133 5839 tablet_bootstrap.cc:492] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.293689 5839 ts_tablet_manager.cc:1367] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240507 08:11:57.294481 5840 ts_tablet_manager.cc:1398] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20240507 08:11:57.294503 5839 raft_consensus.cc:348] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.294705 5839 raft_consensus.cc:374] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.294792 5839 raft_consensus.cc:729] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.295034 5839 consensus_queue.cc:260] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.297219 5258 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_086d5c28a17243c9991e0add9910274a [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 2 got 0
I20240507 08:11:57.297884 5839 ts_tablet_manager.cc:1398] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:57.300089 5547 tablet_service.cc:1467] Processing CreateTablet for tablet b60ec15a83934999871425ef9a64ec72 (DEFAULT_TABLE table=default.Table_aa2b432cc7ef4604b737ade9696fe60f [id=46a6824713664be39257556ade2fe5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.300107 5698 tablet_service.cc:1467] Processing CreateTablet for tablet b60ec15a83934999871425ef9a64ec72 (DEFAULT_TABLE table=default.Table_aa2b432cc7ef4604b737ade9696fe60f [id=46a6824713664be39257556ade2fe5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.300515 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ec15a83934999871425ef9a64ec72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.300537 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ec15a83934999871425ef9a64ec72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.301810 5839 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.305408 5839 tablet_bootstrap.cc:654] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.306342 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4fcc1d7e309946d783a524b61d5f1269
I20240507 08:11:57.306409 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:11:57.306856 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 4fcc1d7e309946d783a524b61d5f1269 deleted and replaced by tablet 59dbc2eaa45d4da58951b45423f284fd
I20240507 08:11:57.307868 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.308055 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_086d5c28a17243c9991e0add9910274a [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 2 got 0
I20240507 08:11:57.308138 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:57.308923 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b27cc38d0c4c4083900cd6765a5cf6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27cc38d0c4c4083900cd6765a5cf6e6 already in progress: creating tablet
W20240507 08:11:57.309252 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4fcc1d7e309946d783a524b61d5f1269 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fcc1d7e309946d783a524b61d5f1269 already in progress: creating tablet
I20240507 08:11:57.309717 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.309957 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.310012 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 67c8624285d946e597227430a5714e2f with delete_type TABLET_DATA_DELETED (replaced by tablet 0d617213b959452ab6519809e56db128) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.310187 5838 ts_tablet_manager.cc:1888] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.310339 5838 log.cc:1196] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f29568ebe4774662a8b49b7829cf30e7
I20240507 08:11:57.310745 5838 ts_tablet_manager.cc:1909] T f29568ebe4774662a8b49b7829cf30e7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.311859 5838 tablet_replica.cc:330] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.312090 5838 raft_consensus.cc:2229] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.312268 5838 raft_consensus.cc:2258] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.312536 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f29568ebe4774662a8b49b7829cf30e7 (table default.Table_a10d53fdc3bc4f009010ffe48156725d [id=4810b964ace44adebd31a95312f8fab9]) successfully deleted
W20240507 08:11:57.312772 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 97c978ec8bca4aa5b2ffaf22c3290747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c978ec8bca4aa5b2ffaf22c3290747 already in progress: creating tablet
I20240507 08:11:57.315027 5839 tablet_bootstrap.cc:492] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:57.316397 5839 ts_tablet_manager.cc:1367] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240507 08:11:57.316617 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.316793 5840 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:57.316856 5837 tablet_replica.cc:330] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.317054 5837 raft_consensus.cc:2229] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.317226 5837 raft_consensus.cc:2258] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.317121 5839 raft_consensus.cc:348] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.317406 5839 raft_consensus.cc:374] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.317502 5839 raft_consensus.cc:729] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.317780 5839 consensus_queue.cc:260] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.318181 5837 ts_tablet_manager.cc:1875] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.320318 5839 ts_tablet_manager.cc:1398] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:11:57.320813 5839 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:57.323704 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 344bf4ab39f74accb383d695578de604 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.324851 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.326540 6240 raft_consensus.cc:482] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.327502 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.327775 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.328173 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccc3e67eeb34c42b36edde7bacd23a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.328567 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e0cbf17e47c943ba8fefb68f7b925e07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.329118 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0cbf17e47c943ba8fefb68f7b925e07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.329818 6240 raft_consensus.cc:504] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.330539 6240 leader_election.cc:288] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.331374 5559 tablet_service.cc:1467] Processing CreateTablet for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6 (DEFAULT_TABLE table=default.Table_641a95550177400e87be665acc80c76c [id=ec447e38f8f34e4a875c1d126a999b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.331532 5839 tablet_bootstrap.cc:654] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.331781 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.333317 5843 tablet_bootstrap.cc:654] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.333782 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4867e1cca1634b4ab6e36f9ed3204e0a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:57.335111 5377 leader_election.cc:341] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:57.335041 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d08cefb6260d4bf499aa464b243906ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.335289 5378 leader_election.cc:341] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:57.335630 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d08cefb6260d4bf499aa464b243906ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.336058 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.336485 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccc3e67eeb34c42b36edde7bacd23a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.337270 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b9fe51f8eb5e42aaa715ab338658f664"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.337844 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9fe51f8eb5e42aaa715ab338658f664 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.341265 5378 leader_election.cc:302] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:57.341956 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.342428 5422 tablet_service.cc:1467] Processing CreateTablet for tablet b60ec15a83934999871425ef9a64ec72 (DEFAULT_TABLE table=default.Table_aa2b432cc7ef4604b737ade9696fe60f [id=46a6824713664be39257556ade2fe5e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.346730 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 156bf04b57724134a45fa9fc477596c5 (DEFAULT_TABLE table=default.Table_3ffbb2e35cc34f288580fefe1e79ebfd [id=004c644ce7ae421f948ccf0fdef44224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.346730 5699 tablet_service.cc:1467] Processing CreateTablet for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6 (DEFAULT_TABLE table=default.Table_641a95550177400e87be665acc80c76c [id=ec447e38f8f34e4a875c1d126a999b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.347133 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156bf04b57724134a45fa9fc477596c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.347775 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.347751 6240 raft_consensus.cc:2726] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:57.347925 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.348176 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ccc3e67eeb34c42b36edde7bacd23a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.349916 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b60ec15a83934999871425ef9a64ec72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.350355 5421 tablet_service.cc:1467] Processing CreateTablet for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6 (DEFAULT_TABLE table=default.Table_641a95550177400e87be665acc80c76c [id=ec447e38f8f34e4a875c1d126a999b27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.350759 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3ed7d7d2e8e424c90048c87c0fb0ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.351717 6240 raft_consensus.cc:482] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.351871 6240 raft_consensus.cc:504] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.352546 6240 leader_election.cc:288] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.353404 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5d566695d94c708e90ee30254084a1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:57.353655 5717 raft_consensus.cc:2454] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:57.354001 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5d566695d94c708e90ee30254084a1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:57.354938 5378 leader_election.cc:341] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.358552 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 156bf04b57724134a45fa9fc477596c5 (DEFAULT_TABLE table=default.Table_3ffbb2e35cc34f288580fefe1e79ebfd [id=004c644ce7ae421f948ccf0fdef44224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.359690 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3a7027841ed1443096999162f8cdd1ea" } modify_external_catalogs: true
I20240507 08:11:57.359957 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3a7027841ed1443096999162f8cdd1ea" } modify_external_catalogs: true
I20240507 08:11:57.363591 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 2b13fbddcab5460db69f43e63f490fba (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.364423 5683 tablet_service.cc:1467] Processing CreateTablet for tablet a5976eca006448e0b9f2573acd738bb1 (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.365334 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156bf04b57724134a45fa9fc477596c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.365525 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5efeda526693400790a6dfb0d118fac0" } modify_external_catalogs: true
I20240507 08:11:57.365726 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5efeda526693400790a6dfb0d118fac0" } modify_external_catalogs: true
I20240507 08:11:57.365999 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b13fbddcab5460db69f43e63f490fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.366497 6073 raft_consensus.cc:482] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.366654 6073 raft_consensus.cc:504] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.367245 6073 leader_election.cc:288] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.367825 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:57.368247 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4010eb808d9c46979560be92d9aa7613" } modify_external_catalogs: true
I20240507 08:11:57.368414 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4010eb808d9c46979560be92d9aa7613" } modify_external_catalogs: true
I20240507 08:11:57.369774 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5976eca006448e0b9f2573acd738bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.369871 5377 leader_election.cc:302] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:57.370364 5995 raft_consensus.cc:482] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.372426 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 2b13fbddcab5460db69f43e63f490fba (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.372779 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b13fbddcab5460db69f43e63f490fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.373683 5425 tablet_service.cc:1467] Processing CreateTablet for tablet a5976eca006448e0b9f2573acd738bb1 (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.374007 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5976eca006448e0b9f2573acd738bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.374771 6240 raft_consensus.cc:2781] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:57.374872 6240 raft_consensus.cc:482] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:57.374955 6240 raft_consensus.cc:3037] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:57.370556 5995 raft_consensus.cc:504] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.378396 5995 leader_election.cc:288] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.379379 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:57.379599 5717 raft_consensus.cc:2454] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:57.380714 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4c064b969844d938fddfe7f043c0b92" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:57.381023 5717 raft_consensus.cc:2379] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fe673739d8084e7da5d7edd722e857d8 in current term 1: Already voted for candidate dc665657d5b34fd7aec884e0f2668e11 in this term.
I20240507 08:11:57.381660 5377 leader_election.cc:302] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:57.381955 5995 raft_consensus.cc:2781] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:57.382045 5995 raft_consensus.cc:482] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:57.382140 5995 raft_consensus.cc:3037] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:57.383708 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:11:57.383930 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:11:57.385686 5995 raft_consensus.cc:504] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.385922 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 47a154769d0f4501837888feba98f33d
I20240507 08:11:57.386907 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a154769d0f4501837888feba98f33d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.388669 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a154769d0f4501837888feba98f33d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.388913 5837 ts_tablet_manager.cc:1888] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.389081 5837 log.cc:1196] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/270a1fd15925416483c211021549315c
I20240507 08:11:57.389312 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 2b13fbddcab5460db69f43e63f490fba (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.389734 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 449d6c8270184ce2918cee8ce731f788 (DEFAULT_TABLE table=default.Table_38e25aba16ae4bcab73abbb11562bfd9 [id=3764a23f5a7847c8b46385bdfaf51f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.390177 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449d6c8270184ce2918cee8ce731f788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.392283 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 449d6c8270184ce2918cee8ce731f788 (DEFAULT_TABLE table=default.Table_38e25aba16ae4bcab73abbb11562bfd9 [id=3764a23f5a7847c8b46385bdfaf51f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.392817 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449d6c8270184ce2918cee8ce731f788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.393847 5683 tablet_service.cc:1467] Processing CreateTablet for tablet eec3704b135740f1a906d60f3ea87ffe (DEFAULT_TABLE table=default.Table_221bfcf09b8e4d5e95e2ab49ad514c10 [id=1345eee5280943ad899d0c66889b16ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.394186 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec3704b135740f1a906d60f3ea87ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.399031 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e14487e625374060b118c9cce4500b3f
I20240507 08:11:57.398988 5423 tablet_service.cc:1467] Processing CreateTablet for tablet eec3704b135740f1a906d60f3ea87ffe (DEFAULT_TABLE table=default.Table_221bfcf09b8e4d5e95e2ab49ad514c10 [id=1345eee5280943ad899d0c66889b16ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.399448 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec3704b135740f1a906d60f3ea87ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.399916 5690 tablet_service.cc:1467] Processing CreateTablet for tablet e7eef600626a4961ab1fae27162537bb (DEFAULT_TABLE table=default.Table_c04dfdb03826409ca727e76221578ea3 [id=e76965f068da4f158275fbed933694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.400310 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eef600626a4961ab1fae27162537bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.405741 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.406038 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.409406 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.409857 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e37b73134cf42ad84b6eec90c9b805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.410177 5837 ts_tablet_manager.cc:1909] T 270a1fd15925416483c211021549315c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:57.410691 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e7eef600626a4961ab1fae27162537bb (DEFAULT_TABLE table=default.Table_c04dfdb03826409ca727e76221578ea3 [id=e76965f068da4f158275fbed933694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.411175 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eef600626a4961ab1fae27162537bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.413193 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 270a1fd15925416483c211021549315c (table default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]) successfully deleted
I20240507 08:11:57.413417 5260 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 17f8fa3593a141c5a56d6ed7d59cc4dd
W20240507 08:11:57.413437 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 47a154769d0f4501837888feba98f33d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47a154769d0f4501837888feba98f33d already in progress: creating tablet
W20240507 08:11:57.413923 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e14487e625374060b118c9cce4500b3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14487e625374060b118c9cce4500b3f already in progress: creating tablet
I20240507 08:11:57.410691 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.414439 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e37b73134cf42ad84b6eec90c9b805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.417671 6308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.253:43459 every 8 connection(s)
I20240507 08:11:57.418021 5940 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.253:43459
I20240507 08:11:57.418372 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.418756 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fce98f472ef46589dbea6b6aefa6572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.420094 5940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
I20240507 08:11:57.421317 5840 tablet_bootstrap.cc:654] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.421905 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.422286 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fce98f472ef46589dbea6b6aefa6572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.422927 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7f072f55a94cca938aa063637af9a5 (DEFAULT_TABLE table=default.Table_a778d8595fc7496bace57c63194455fe [id=525d572835834c26911e204eb69e8d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.423271 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7f072f55a94cca938aa063637af9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.424875 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7f072f55a94cca938aa063637af9a5 (DEFAULT_TABLE table=default.Table_a778d8595fc7496bace57c63194455fe [id=525d572835834c26911e204eb69e8d90]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.425220 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7f072f55a94cca938aa063637af9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.429186 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 5940
I20240507 08:11:57.432869 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 6def4bf6a7554a4090a39b1823c4097c (DEFAULT_TABLE table=default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf [id=a57d9cd5e2d5498f9ae045b87c5ccb79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.433205 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def4bf6a7554a4090a39b1823c4097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.433545 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_97f2cc68bfaf4e558499da922503bdb1" } new_table_name: "default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6" modify_external_catalogs: true
I20240507 08:11:57.433643 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b949ed5c663541aab24940387df46388"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.434149 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b949ed5c663541aab24940387df46388 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.435204 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_457ae0dd8b974005872846c9fe575b10" } modify_external_catalogs: true
I20240507 08:11:57.435388 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_457ae0dd8b974005872846c9fe575b10" } modify_external_catalogs: true
I20240507 08:11:57.436579 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_55ca0ba456d0447d95c4f2df7eee0591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.437510 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.437944 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0b399a2d074f4ca3626ff17bd5518e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:57.440333 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55ca0ba456d0447d95c4f2df7eee0591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.442499 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff69b3775d94e5da1f9f327c759abb8 (DEFAULT_TABLE table=default.Table_7c0ab3c1b77745b796fe0a735b7bc292 [id=401f3776823e4393812332c4c093561c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.442761 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca (DEFAULT_TABLE table=default.Table_18ef77bde60b44179d9367054bffc7af [id=b40ed3eac5e4406299c3ad2934df75db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.443173 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff69b3775d94e5da1f9f327c759abb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.446264 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a94c97ecc5d47abb190f3833fd317ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.446187 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.447466 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0b399a2d074f4ca3626ff17bd5518e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.448426 5689 tablet_service.cc:1467] Processing CreateTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a (DEFAULT_TABLE table=default.Table_28dc4a4ae86d46e79217cc266d671289 [id=0f762f862f334dcdaf39bad447d697c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.450060 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 6def4bf6a7554a4090a39b1823c4097c (DEFAULT_TABLE table=default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf [id=a57d9cd5e2d5498f9ae045b87c5ccb79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.450428 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def4bf6a7554a4090a39b1823c4097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.452775 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca (DEFAULT_TABLE table=default.Table_18ef77bde60b44179d9367054bffc7af [id=b40ed3eac5e4406299c3ad2934df75db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.456815 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.457654 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a94c97ecc5d47abb190f3833fd317ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.462030 5841 ts_tablet_manager.cc:1888] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.462224 5841 log.cc:1196] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/270a1fd15925416483c211021549315c
I20240507 08:11:57.462662 5841 ts_tablet_manager.cc:1909] T 270a1fd15925416483c211021549315c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:57.463569 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 156bf04b57724134a45fa9fc477596c5 (DEFAULT_TABLE table=default.Table_3ffbb2e35cc34f288580fefe1e79ebfd [id=004c644ce7ae421f948ccf0fdef44224]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.463737 5841 tablet_replica.cc:330] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:57.464005 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156bf04b57724134a45fa9fc477596c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.464242 5995 leader_election.cc:288] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.466653 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:57.467128 5717 raft_consensus.cc:3037] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:57.467809 5716 raft_consensus.cc:1261] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:57.470412 5717 raft_consensus.cc:2454] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:57.471015 5377 leader_election.cc:302] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:11:57.481731 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d617213b959452ab6519809e56db128 consensus state has no leader
I20240507 08:11:57.487918 5691 tablet_service.cc:1467] Processing CreateTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 (DEFAULT_TABLE table=default.Table_615d90859dd74c9aa813e233b63dd68e [id=0652c43a334e4f04a34ff59ca43e545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.488458 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee9e434083f4fa6859cfe2b7cca16e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.488827 5995 consensus_queue.cc:1035] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:57.489681 5995 raft_consensus.cc:2781] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:57.490422 5995 raft_consensus.cc:686] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:57.490901 5995 consensus_queue.cc:237] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.493723 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:57.494239 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b13fbddcab5460db69f43e63f490fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.498554 6304 raft_consensus.cc:482] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.499306 6304 raft_consensus.cc:504] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.500797 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 270a1fd15925416483c211021549315c (table default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]) successfully deleted
W20240507 08:11:57.501099 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8a015bf84a454d49b6497c16ed15c84a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a015bf84a454d49b6497c16ed15c84a already in progress: creating tablet
I20240507 08:11:57.499660 6240 raft_consensus.cc:504] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.503926 6304 leader_election.cc:288] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.504072 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf34089e5ec419f97c0da9ec42d7eec" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.504478 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 0d617213b959452ab6519809e56db128 (table default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6 [id=bce0bd66e7b74db7934937410c81b20b], current schema version=5) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0d617213b959452ab6519809e56db128 consensus state has no leader
I20240507 08:11:57.504400 6240 leader_election.cc:288] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.504803 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e9691d8ea6744056afe858914bac70c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.506891 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9691d8ea6744056afe858914bac70c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.508011 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c04dfdb03826409ca727e76221578ea3" } new_table_name: "default.Table_c76e55c0da46460a8ad96f8f966b29e7" modify_external_catalogs: true
I20240507 08:11:57.510053 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 565ac6091f5e48b1b024094d5560e275
I20240507 08:11:57.510696 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f10e0f30c1f14fa18ead66c614f8473e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.511448 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f10e0f30c1f14fa18ead66c614f8473e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:57.512784 5377 leader_election.cc:341] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.515039 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5d566695d94c708e90ee30254084a1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:57.515240 5716 raft_consensus.cc:3037] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:57.518913 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bc5e572b5c614e0fbb526ab59a22254f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.519369 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc5e572b5c614e0fbb526ab59a22254f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.522341 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_12317138695c42d695226bd546e857ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.521726 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_66abcf77e5b04c1ea97c7079c91bfada" } new_table_name: "default.Table_4230ce43f9b045a3a4fe583a727d0063" modify_external_catalogs: true
I20240507 08:11:57.525607 5411 tablet_service.cc:1467] Processing CreateTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 (DEFAULT_TABLE table=default.Table_615d90859dd74c9aa813e233b63dd68e [id=0652c43a334e4f04a34ff59ca43e545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.526108 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee9e434083f4fa6859cfe2b7cca16e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.529816 5413 tablet_service.cc:1467] Processing CreateTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a (DEFAULT_TABLE table=default.Table_28dc4a4ae86d46e79217cc266d671289 [id=0f762f862f334dcdaf39bad447d697c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.530344 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.531757 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_086d5c28a17243c9991e0add9910274a" } new_table_name: "default.Table_e5036f51d41048c1bd6c65fbb8298939" modify_external_catalogs: true
I20240507 08:11:57.532104 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f4ad9488db7b43469ea4ec3d4e365ba4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.532546 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4ad9488db7b43469ea4ec3d4e365ba4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:57.533979 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12317138695c42d695226bd546e857ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.529029 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_54adb6f25ed342928404f4a1853c9a60" } new_table_name: "default.Table_e5179c5e5c224b3690b690b725a01cd0" modify_external_catalogs: true
W20240507 08:11:57.539057 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7eef600626a4961ab1fae27162537bb consensus state has no leader
I20240507 08:11:57.539300 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c290c4af761c45e885f1f4af9a48f4fc" } modify_external_catalogs: true
I20240507 08:11:57.547480 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c290c4af761c45e885f1f4af9a48f4fc" } modify_external_catalogs: true
I20240507 08:11:57.550663 5995 raft_consensus.cc:482] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.550966 5995 raft_consensus.cc:504] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.551432 5716 raft_consensus.cc:2454] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:57.552299 5377 leader_election.cc:302] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:57.552703 6240 raft_consensus.cc:2781] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:57.552874 6240 raft_consensus.cc:686] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:57.553110 6240 consensus_queue.cc:237] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.463982 5841 raft_consensus.cc:2229] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.556849 5841 raft_consensus.cc:2258] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.557861 5841 ts_tablet_manager.cc:1875] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.559960 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_14fe89e6f91f4d1593dde48abd8bebda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.560470 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14fe89e6f91f4d1593dde48abd8bebda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.564381 5995 leader_election.cc:288] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.565026 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6670e99e4a4442801d42950eb7ab79" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.566018 5377 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.569520 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_115a2eafb15744289ae668ac98becc7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.569818 5843 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.569977 5843 ts_tablet_manager.cc:1367] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.305s user 0.005s sys 0.001s
W20240507 08:11:57.570094 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_115a2eafb15744289ae668ac98becc7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.571125 5843 raft_consensus.cc:348] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.571420 5843 raft_consensus.cc:374] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.571539 5843 raft_consensus.cc:729] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.571815 5843 consensus_queue.cc:260] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.571991 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f993482dcb444626a4d006bf1755dcca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.572551 5843 ts_tablet_manager.cc:1398] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20240507 08:11:57.572588 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f993482dcb444626a4d006bf1755dcca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.572935 5843 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:11:57.574790 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43225934cef04109b3dbc057b230f032 consensus state has no leader
I20240507 08:11:57.575480 5843 tablet_bootstrap.cc:654] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.578997 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7d89e4185229431f9a68f599da4aff34
I20240507 08:11:57.585059 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d89e4185229431f9a68f599da4aff34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:57.585191 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03a59c7257e74b4abaa4ea7a1ee9136d consensus state has no leader
W20240507 08:11:57.585880 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7d89e4185229431f9a68f599da4aff34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d89e4185229431f9a68f599da4aff34 already in progress: creating tablet
I20240507 08:11:57.589993 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b949ed5c663541aab24940387df46388" } modify_external_catalogs: true
I20240507 08:11:57.590266 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b949ed5c663541aab24940387df46388" } modify_external_catalogs: true
I20240507 08:11:57.601488 5257 catalog_manager.cc:5484] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:57.602346 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_0856317e9911474aa5861773d7185325 [id=0a1bc06a351549d5aea075ad8d026476]). Expected version 2 got 0
I20240507 08:11:57.602695 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 3 got 0
I20240507 08:11:57.607223 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fcf86fbc38ac4137b5b9acb214330443" } modify_external_catalogs: true
I20240507 08:11:57.607492 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fcf86fbc38ac4137b5b9acb214330443" } modify_external_catalogs: true
I20240507 08:11:57.613118 6309 raft_consensus.cc:482] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.613248 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d89e4185229431f9a68f599da4aff34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.613301 6309 raft_consensus.cc:504] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.614200 6309 leader_election.cc:288] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.617719 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.618137 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_18ef77bde60b44179d9367054bffc7af" } modify_external_catalogs: true
I20240507 08:11:57.618423 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_18ef77bde60b44179d9367054bffc7af" } modify_external_catalogs: true
W20240507 08:11:57.618669 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 565ac6091f5e48b1b024094d5560e275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565ac6091f5e48b1b024094d5560e275 already in progress: creating tablet
I20240507 08:11:57.620323 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed8bac124cb4de983f0fdbaf728975f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.621593 5377 leader_election.cc:341] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.629055 5843 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.629317 5843 ts_tablet_manager.cc:1367] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.002s
I20240507 08:11:57.631695 5843 raft_consensus.cc:348] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.633903 5843 raft_consensus.cc:374] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.634552 5843 raft_consensus.cc:729] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.635156 5843 consensus_queue.cc:260] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.635577 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 (DEFAULT_TABLE table=default.Table_c2d5a28991f9466486b34132cbd61131 [id=11df3920eb394624a4d2e85471cba9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.636181 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ccdad494794192b3c37c8d5cbfcb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.636696 6325 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:57.644780 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 43225934cef04109b3dbc057b230f032 (DEFAULT_TABLE table=default.Table_4230ce43f9b045a3a4fe583a727d0063 [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.645071 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 (DEFAULT_TABLE table=default.Table_c2d5a28991f9466486b34132cbd61131 [id=11df3920eb394624a4d2e85471cba9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.645200 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43225934cef04109b3dbc057b230f032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.645443 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ccdad494794192b3c37c8d5cbfcb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.652354 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 66247e08dfdf47f69ba24b074c8080f0 (DEFAULT_TABLE table=default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.652925 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66247e08dfdf47f69ba24b074c8080f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.655103 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 43225934cef04109b3dbc057b230f032 (DEFAULT_TABLE table=default.Table_4230ce43f9b045a3a4fe583a727d0063 [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.655548 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43225934cef04109b3dbc057b230f032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.655457 5260 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 959a36625a754d0dab748730216d5e76
I20240507 08:11:57.659018 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 66247e08dfdf47f69ba24b074c8080f0 (DEFAULT_TABLE table=default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.660642 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66247e08dfdf47f69ba24b074c8080f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.661624 5549 tablet_service.cc:1467] Processing CreateTablet for tablet a5976eca006448e0b9f2573acd738bb1 (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.661932 5838 ts_tablet_manager.cc:1875] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.663584 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069517174093 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:11:57.663841 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:11:57.665953 6073 raft_consensus.cc:482] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.666112 6073 raft_consensus.cc:504] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.666361 5686 tablet_service.cc:1467] Processing CreateTablet for tablet a605be6967a041388dfe81053fbb602e (DEFAULT_TABLE table=default.Table_ed36193e2b664d80b2a509e4f94623bd [id=b92037dbac744552b58099b7fdc424b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.666887 6073 leader_election.cc:288] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.640513 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.676128 5843 ts_tablet_manager.cc:1398] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20240507 08:11:57.676530 5843 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.680876 5841 ts_tablet_manager.cc:1888] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.685482 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3a94c97ecc5d47abb190f3833fd317ca
I20240507 08:11:57.686587 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 049483b0533642f2b46d8fa0e9b123cc
I20240507 08:11:57.687954 6073 raft_consensus.cc:482] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.688853 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ed36193e2b664d80b2a509e4f94623bd" } new_table_name: "default.Table_13bfcb3083e54a37b5fad286ae0e5b26" modify_external_catalogs: true
W20240507 08:11:57.689455 5520 leader_election.cc:341] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 344bf4ab39f74accb383d695578de604
I20240507 08:11:57.689909 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 449d6c8270184ce2918cee8ce731f788 (DEFAULT_TABLE table=default.Table_38e25aba16ae4bcab73abbb11562bfd9 [id=3764a23f5a7847c8b46385bdfaf51f20]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.690124 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a154769d0f4501837888feba98f33d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.693773 5841 log.cc:1196] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:57.693890 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5976eca006448e0b9f2573acd738bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.694370 5841 ts_tablet_manager.cc:1909] T 4867e1cca1634b4ab6e36f9ed3204e0a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:57.691526 5843 tablet_bootstrap.cc:654] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.697816 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 449d6c8270184ce2918cee8ce731f788. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.699143 5841 tablet_replica.cc:330] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:57.695163 6073 raft_consensus.cc:504] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.699371 5841 raft_consensus.cc:2229] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.699566 5841 raft_consensus.cc:2258] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.700093 5841 ts_tablet_manager.cc:1875] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:57.701515 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a605be6967a041388dfe81053fbb602e consensus state has no leader
I20240507 08:11:57.701707 5561 tablet_service.cc:1467] Processing CreateTablet for tablet eec3704b135740f1a906d60f3ea87ffe (DEFAULT_TABLE table=default.Table_221bfcf09b8e4d5e95e2ab49ad514c10 [id=1345eee5280943ad899d0c66889b16ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.705847 6317 raft_consensus.cc:482] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.706122 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605be6967a041388dfe81053fbb602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.706246 6317 raft_consensus.cc:504] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.707366 6317 leader_election.cc:288] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.712273 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4867e1cca1634b4ab6e36f9ed3204e0a (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
W20240507 08:11:57.712522 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 97c978ec8bca4aa5b2ffaf22c3290747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97c978ec8bca4aa5b2ffaf22c3290747 already in progress: creating tablet
W20240507 08:11:57.712747 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b1fadbfd330b4e699e151f83125da281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fadbfd330b4e699e151f83125da281 already in progress: creating tablet
W20240507 08:11:57.712710 5520 leader_election.cc:341] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: ea17889619fa420fbeacf1b41594c598
W20240507 08:11:57.712937 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c839c8a5041f4c3da9bc0749afee1413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c839c8a5041f4c3da9bc0749afee1413 already in progress: creating tablet
W20240507 08:11:57.713166 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4e63c3f3ecc84913ad05864be450e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e63c3f3ecc84913ad05864be450e9b0 already in progress: creating tablet
I20240507 08:11:57.714788 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 3 got 0
I20240507 08:11:57.715294 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:57.715544 6309 consensus_queue.cc:1035] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:57.716291 5425 tablet_service.cc:1467] Processing CreateTablet for tablet a605be6967a041388dfe81053fbb602e (DEFAULT_TABLE table=default.Table_ed36193e2b664d80b2a509e4f94623bd [id=b92037dbac744552b58099b7fdc424b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.716984 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605be6967a041388dfe81053fbb602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.720175 6325 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:57.720160 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_46b030f735704339a1921f803d7af17d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.720791 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:57.721130 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff69b3775d94e5da1f9f327c759abb8 (DEFAULT_TABLE table=default.Table_7c0ab3c1b77745b796fe0a735b7bc292 [id=401f3776823e4393812332c4c093561c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.721257 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:57.721634 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46b030f735704339a1921f803d7af17d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.722932 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cb5f53bbcf3049939314d98e891fe83c" } modify_external_catalogs: true
I20240507 08:11:57.723219 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cb5f53bbcf3049939314d98e891fe83c" } modify_external_catalogs: true
I20240507 08:11:57.724982 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191adef34efa4c3999d4642a80177294" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.725495 5377 leader_election.cc:341] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.727347 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_15e48d810d164ffba2efc7a11fc51244"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.727892 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15e48d810d164ffba2efc7a11fc51244 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.730618 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_94c6194476954481ab568218481b03c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.731102 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94c6194476954481ab568218481b03c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.731436 6311 sys_catalog.cc:263] Verifying existing consensus state
I20240507 08:11:57.733904 5843 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.734067 5843 ts_tablet_manager.cc:1367] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.002s
I20240507 08:11:57.734917 5843 raft_consensus.cc:348] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.735162 5843 raft_consensus.cc:374] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.735278 5843 raft_consensus.cc:729] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.735571 5843 consensus_queue.cc:260] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.744716 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d9766986d2d04989a90881e64f84e78e" } new_table_name: "default.Table_a843afde2c19414c9b6c0581df4cb832" modify_external_catalogs: true
I20240507 08:11:57.745922 5559 tablet_service.cc:1467] Processing CreateTablet for tablet e7eef600626a4961ab1fae27162537bb (DEFAULT_TABLE table=default.Table_c04dfdb03826409ca727e76221578ea3 [id=e76965f068da4f158275fbed933694ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.746567 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cec30b6d6641468b8bb09f32c9162560"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.746714 5843 ts_tablet_manager.cc:1398] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
W20240507 08:11:57.747056 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cec30b6d6641468b8bb09f32c9162560 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.748878 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e1b850d9c6864653b0ca7439322d02a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.749332 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1b850d9c6864653b0ca7439322d02a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.750865 5843 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.751462 6327 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:57.752023 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff69b3775d94e5da1f9f327c759abb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.752494 6317 raft_consensus.cc:482] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.752664 6317 raft_consensus.cc:504] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.753387 6317 leader_election.cc:288] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.753983 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a2cfb5f046434c8340234a447a7e65" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.754474 5377 leader_election.cc:341] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.755355 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f (DEFAULT_TABLE table=default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.755725 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eec3704b135740f1a906d60f3ea87ffe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.755321 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.756548 6073 leader_election.cc:288] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:57.756729 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 049483b0533642f2b46d8fa0e9b123cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 049483b0533642f2b46d8fa0e9b123cc already in progress: creating tablet
I20240507 08:11:57.757803 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:57.759100 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3a94c97ecc5d47abb190f3833fd317ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a94c97ecc5d47abb190f3833fd317ca already in progress: creating tablet
I20240507 08:11:57.761129 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e37b73134cf42ad84b6eec90c9b805f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.766134 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.766331 5843 tablet_bootstrap.cc:654] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.766606 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7eef600626a4961ab1fae27162537bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.767917 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 (DEFAULT_TABLE table=default.Table_4fa2d0d8a1a9482599a4d430029b2b4b [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.768981 6317 raft_consensus.cc:482] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.769151 6317 raft_consensus.cc:504] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.769920 6317 leader_election.cc:288] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.770236 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 3c7f072f55a94cca938aa063637af9a5 (DEFAULT_TABLE table=default.Table_a778d8595fc7496bace57c63194455fe [id=525d572835834c26911e204eb69e8d90]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:57.770862 5377 leader_election.cc:341] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 8e3ffaf1781e4b108f305346476905be
I20240507 08:11:57.773095 5840 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.773247 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:57.773499 5840 ts_tablet_manager.cc:1367] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.457s user 0.006s sys 0.002s
I20240507 08:11:57.777734 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f69f572cb6f6424eb22ea998d9f6ba3f" } new_table_name: "default.Table_6e265b95a5364ac98a18cb1a4b2b26f1" modify_external_catalogs: true
I20240507 08:11:57.779994 5840 raft_consensus.cc:348] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.785825 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d08cefb6260d4bf499aa464b243906ac" } modify_external_catalogs: true
I20240507 08:11:57.785919 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fce98f472ef46589dbea6b6aefa6572. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.786062 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d08cefb6260d4bf499aa464b243906ac" } modify_external_catalogs: true
I20240507 08:11:57.787817 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d89e4185229431f9a68f599da4aff34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.788041 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.788259 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e (DEFAULT_TABLE table=default.Table_2ee2e89703be4de1a6f05d13e2af9abb [id=0a13053ee3a44ee1a9588eb3ba56b74f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:57.788597 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.788748 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.789052 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 6def4bf6a7554a4090a39b1823c4097c (DEFAULT_TABLE table=default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf [id=a57d9cd5e2d5498f9ae045b87c5ccb79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.794840 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap starting.
I20240507 08:11:57.795951 5840 raft_consensus.cc:374] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.796051 5840 raft_consensus.cc:729] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.799120 5546 tablet_service.cc:1467] Processing CreateTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 (DEFAULT_TABLE table=default.Table_615d90859dd74c9aa813e233b63dd68e [id=0652c43a334e4f04a34ff59ca43e545a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.799647 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca (DEFAULT_TABLE table=default.Table_18ef77bde60b44179d9367054bffc7af [id=b40ed3eac5e4406299c3ad2934df75db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.802821 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f0ed0e76d4c746c189b43ebcf1c271b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.803438 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0ed0e76d4c746c189b43ebcf1c271b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:57.805891 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34adee00b529424b9f274c61ce73b0bc consensus state has no leader
I20240507 08:11:57.806375 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_90a87f2fadb249189e0eb022d3730409" } new_table_name: "default.Table_9342125a817c439eb044fbe82437c21a" modify_external_catalogs: true
I20240507 08:11:57.809439 5260 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 817542fbc9624497b604b0bc2e39340a
I20240507 08:11:57.812657 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3d84936f1b42408a983ad41684136fb3" } new_table_name: "default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a" modify_external_catalogs: true
I20240507 08:11:57.821599 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 43225934cef04109b3dbc057b230f032 (DEFAULT_TABLE table=default.Table_4230ce43f9b045a3a4fe583a727d0063 [id=58282ffddcba4e4db7736a325a4a530a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.822921 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c7f072f55a94cca938aa063637af9a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.823297 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:57.830018 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43225934cef04109b3dbc057b230f032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.830127 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 3 got 2
I20240507 08:11:57.832136 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 66247e08dfdf47f69ba24b074c8080f0 (DEFAULT_TABLE table=default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.832665 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f0b399a2d074f4ca3626ff17bd5518e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.833140 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bee9e434083f4fa6859cfe2b7cca16e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.834097 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a94c97ecc5d47abb190f3833fd317ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.835680 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6def4bf6a7554a4090a39b1823c4097c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.840655 5562 tablet_service.cc:1467] Processing CreateTablet for tablet a605be6967a041388dfe81053fbb602e (DEFAULT_TABLE table=default.Table_ed36193e2b664d80b2a509e4f94623bd [id=b92037dbac744552b58099b7fdc424b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.842339 5255 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:57.844360 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_67202ebc0080457d8ccacb243261511e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:57.844746 6311 log.cc:824] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Log is configured to *not* fsync() on all Append() calls
W20240507 08:11:57.844969 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67202ebc0080457d8ccacb243261511e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.846774 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ad303c5560534112a84810140d744045"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.847594 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad303c5560534112a84810140d744045 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.850931 5840 consensus_queue.cc:260] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.852257 5555 tablet_service.cc:1467] Processing CreateTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a (DEFAULT_TABLE table=default.Table_28dc4a4ae86d46e79217cc266d671289 [id=0f762f862f334dcdaf39bad447d697c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.787586 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 (DEFAULT_TABLE table=default.Table_c2d5a28991f9466486b34132cbd61131 [id=11df3920eb394624a4d2e85471cba9d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.800002 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 5ff69b3775d94e5da1f9f327c759abb8 (DEFAULT_TABLE table=default.Table_7c0ab3c1b77745b796fe0a735b7bc292 [id=401f3776823e4393812332c4c093561c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:57.863435 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_269bbea2c2e04830a74ce31662db3c24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.864323 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_269bbea2c2e04830a74ce31662db3c24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.866930 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.875331 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a605be6967a041388dfe81053fbb602e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.875617 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66247e08dfdf47f69ba24b074c8080f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:57.876408 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8ed8bac124cb4de983f0fdbaf728975f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed8bac124cb4de983f0fdbaf728975f already in progress: creating tablet
I20240507 08:11:57.877413 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63ccdad494794192b3c37c8d5cbfcb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.878173 5263 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet f9678c683c4f4b5baa55c45676c70d90
I20240507 08:11:57.873047 6327 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:57.878598 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_221bfcf09b8e4d5e95e2ab49ad514c10" } modify_external_catalogs: true
I20240507 08:11:57.878877 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_221bfcf09b8e4d5e95e2ab49ad514c10" } modify_external_catalogs: true
I20240507 08:11:57.881633 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 5405f264bac1464a8f1b4435715a64a4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:57.882445 5840 ts_tablet_manager.cc:1398] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.106s user 0.002s sys 0.000s
I20240507 08:11:57.884594 5840 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:57.885272 5378 leader_election.cc:341] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.886978 6309 consensus_queue.cc:1035] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:57.887257 5840 tablet_bootstrap.cc:654] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.902578 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.903985 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.904078 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:57.907354 5517 leader_election.cc:302] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:11:57.913246 5517 leader_election.cc:341] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: ea17889619fa420fbeacf1b41594c598
I20240507 08:11:57.913439 5517 leader_election.cc:302] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:57.916954 6073 raft_consensus.cc:3037] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:57.927567 5838 ts_tablet_manager.cc:1888] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.930371 5840 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:57.930656 5840 ts_tablet_manager.cc:1367] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.000s
I20240507 08:11:57.931480 5840 raft_consensus.cc:348] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.931751 5840 raft_consensus.cc:374] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.931871 5840 raft_consensus.cc:729] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.932155 5840 consensus_queue.cc:260] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.932878 5840 ts_tablet_manager.cc:1398] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:11:57.935104 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ff69b3775d94e5da1f9f327c759abb8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:57.936550 6240 raft_consensus.cc:482] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.936784 6240 raft_consensus.cc:504] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:57.938063 6240 leader_election.cc:288] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:57.938165 5843 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:57.938349 5843 ts_tablet_manager.cc:1367] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.188s user 0.037s sys 0.057s
I20240507 08:11:57.938951 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bc7bc06e494adcb6a8641b50b90e6c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:57.939538 5377 leader_election.cc:341] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.939669 5843 raft_consensus.cc:348] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.939920 5843 raft_consensus.cc:374] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:57.940008 5843 raft_consensus.cc:729] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:57.940392 5843 consensus_queue.cc:260] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:57.940728 6337 raft_consensus.cc:2726] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:57.940851 5517 leader_election.cc:341] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 344bf4ab39f74accb383d695578de604
I20240507 08:11:57.941016 6337 raft_consensus.cc:482] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:57.941433 5843 ts_tablet_manager.cc:1398] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:11:57.941808 5843 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:57.943583 5843 tablet_bootstrap.cc:654] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.941028 5517 leader_election.cc:302] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:57.947907 5841 ts_tablet_manager.cc:1888] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:57.948705 5841 log.cc:1196] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/bc619db3a2944ecc9f0112b28de148a6
I20240507 08:11:57.950353 5838 log.cc:1196] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5ca8fc2c7f3849c380fea56862ffc26e
I20240507 08:11:57.952425 5840 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:57.954139 5841 ts_tablet_manager.cc:1909] T bc619db3a2944ecc9f0112b28de148a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:57.955978 6338 raft_consensus.cc:2726] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:57.956722 5840 tablet_bootstrap.cc:654] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:57.959746 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b1add7221be940138ee5b9e2e796e92c" } modify_external_catalogs: true
I20240507 08:11:57.959968 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b1add7221be940138ee5b9e2e796e92c" } modify_external_catalogs: true
I20240507 08:11:57.963980 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:57.966634 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.966949 5838 ts_tablet_manager.cc:1909] T 5ca8fc2c7f3849c380fea56862ffc26e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:57.966997 5837 tablet_replica.cc:330] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:57.967265 5837 raft_consensus.cc:2229] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:57.967517 5837 raft_consensus.cc:2258] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.967757 5838 tablet_replica.cc:330] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:57.967962 5838 raft_consensus.cc:2229] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:57.968333 5838 raft_consensus.cc:2258] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:57.968906 5838 ts_tablet_manager.cc:1875] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:57.970050 5837 ts_tablet_manager.cc:1875] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:57.971879 5839 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.105s sys 0.135s
I20240507 08:11:57.972461 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
W20240507 08:11:57.975956 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e5439844f384513b4719cfe4f4e5d9b consensus state has no leader
I20240507 08:11:57.978135 5262 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_6e265b95a5364ac98a18cb1a4b2b26f1 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 2 got 0
I20240507 08:11:57.979189 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8ebdb9219e974871b5a4f78b6abfd78f" } new_table_name: "default.Table_6ad0c511f95b4b09abad3bc4a13d1238" modify_external_catalogs: true
I20240507 08:11:57.979343 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:57.979404 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:57.981420 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43644b4405f44bfd9a1afcaedbd68db5 consensus state has no leader
I20240507 08:11:57.981993 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c2d5a28991f9466486b34132cbd61131" } modify_external_catalogs: true
I20240507 08:11:57.982168 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c2d5a28991f9466486b34132cbd61131" } modify_external_catalogs: true
I20240507 08:11:57.984406 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_61d23e54efcc415c9f29270b3e622bc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:57.984989 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61d23e54efcc415c9f29270b3e622bc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:57.988934 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6670e99e4a4442801d42950eb7ab79" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:57.990361 5839 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.640s user 0.105s sys 0.135s
I20240507 08:11:57.992244 6325 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:57.997932 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bc7bc06e494adcb6a8641b50b90e6c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:57.998481 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5ca8fc2c7f3849c380fea56862ffc26e (table default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7 [id=c5ee2823c21745f380337f43009224f7]) successfully deleted
I20240507 08:11:57.998631 5837 ts_tablet_manager.cc:1888] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:57.998768 5837 log.cc:1196] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8ed8bac124cb4de983f0fdbaf728975f
W20240507 08:11:57.998769 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b1fadbfd330b4e699e151f83125da281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b1fadbfd330b4e699e151f83125da281 already in progress: creating tablet
W20240507 08:11:57.998996 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c839c8a5041f4c3da9bc0749afee1413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c839c8a5041f4c3da9bc0749afee1413 already in progress: creating tablet
I20240507 08:11:57.999243 5837 ts_tablet_manager.cc:1909] T 8ed8bac124cb4de983f0fdbaf728975f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:11:57.999269 5378 leader_election.cc:341] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:57.999720 5378 consensus_peers.cc:485] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:11:57.999883 5378 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:57.999991 5378 leader_election.cc:302] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.000488 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8ed8bac124cb4de983f0fdbaf728975f (table default.Table_71f59cb7918c42d29419261d5f361830 [id=2e71cf49719f4d9fb07c289e1428080c]) successfully deleted
W20240507 08:11:58.000545 5378 leader_election.cc:341] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.000666 5378 leader_election.cc:302] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.001016 6240 raft_consensus.cc:2726] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.001361 6240 consensus_queue.cc:1035] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:58.001809 6317 raft_consensus.cc:2726] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.001361 6337 raft_consensus.cc:504] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.006147 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5d566695d94c708e90ee30254084a1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:58.006665 5378 leader_election.cc:341] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.007725 5837 tablet_replica.cc:330] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:58.008554 5837 raft_consensus.cc:2229] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.008960 5837 raft_consensus.cc:2258] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.009155 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88a2cfb5f046434c8340234a447a7e65" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.011471 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf34089e5ec419f97c0da9ec42d7eec" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:58.012428 5378 leader_election.cc:341] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.012554 5378 leader_election.cc:302] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:58.013451 5378 leader_election.cc:341] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.013710 5378 leader_election.cc:302] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.014123 6304 raft_consensus.cc:2726] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.013979 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_edfefd7112b84facba5e8f004b5c1c1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:58.013911 6240 raft_consensus.cc:2726] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:58.014729 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edfefd7112b84facba5e8f004b5c1c1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.015326 5840 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.015487 5840 ts_tablet_manager.cc:1367] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.063s user 0.007s sys 0.002s
I20240507 08:11:58.016310 5841 tablet_replica.cc:330] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.016238 5840 raft_consensus.cc:348] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.016512 5841 raft_consensus.cc:2229] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.016652 5841 raft_consensus.cc:2258] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.017174 5841 ts_tablet_manager.cc:1875] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.018060 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8ed8bac124cb4de983f0fdbaf728975f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.018314 5840 raft_consensus.cc:374] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.018631 5840 raft_consensus.cc:729] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.019191 5840 consensus_queue.cc:260] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.020452 6341 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:58.025061 5840 ts_tablet_manager.cc:1398] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:11:58.025982 5840 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.027254 5837 ts_tablet_manager.cc:1875] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.027873 5843 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:58.028023 5843 ts_tablet_manager.cc:1367] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.086s user 0.006s sys 0.026s
I20240507 08:11:58.028751 5843 raft_consensus.cc:348] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.028950 5843 raft_consensus.cc:374] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.029060 5843 raft_consensus.cc:729] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.029346 5843 consensus_queue.cc:260] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.029893 5843 ts_tablet_manager.cc:1398] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:58.030179 5843 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:58.031471 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=20 overwritten=0 applied=20 ignored=0} inserts{seen=23 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:11:58.032114 6311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap complete.
I20240507 08:11:58.032243 5839 tablet_bootstrap.cc:492] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:58.032380 5839 ts_tablet_manager.cc:1367] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.712s user 0.106s sys 0.136s
I20240507 08:11:58.033638 5841 ts_tablet_manager.cc:1888] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.017540 6337 leader_election.cc:288] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.033778 5841 log.cc:1196] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/6bf34089e5ec419f97c0da9ec42d7eec
I20240507 08:11:58.034127 5841 ts_tablet_manager.cc:1909] T 6bf34089e5ec419f97c0da9ec42d7eec P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.035194 5841 tablet_replica.cc:330] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.035128 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 6bf34089e5ec419f97c0da9ec42d7eec (table default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]) successfully deleted
I20240507 08:11:58.035380 5841 raft_consensus.cc:2229] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.035542 5841 raft_consensus.cc:2258] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:58.036679 5520 leader_election.cc:341] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 67c8624285d946e597227430a5714e2f
I20240507 08:11:58.037436 5841 ts_tablet_manager.cc:1875] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.038061 5718 raft_consensus.cc:1261] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Refusing update from remote peer dc665657d5b34fd7aec884e0f2668e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:11:58.038851 6304 consensus_queue.cc:1035] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:58.039187 5843 tablet_bootstrap.cc:654] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.045492 5841 ts_tablet_manager.cc:1888] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.045622 5841 log.cc:1196] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/6f6670e99e4a4442801d42950eb7ab79
I20240507 08:11:58.045930 5841 ts_tablet_manager.cc:1909] T 6f6670e99e4a4442801d42950eb7ab79 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.048261 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:58.049438 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 6f6670e99e4a4442801d42950eb7ab79 (table default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]) successfully deleted
I20240507 08:11:58.049887 5841 tablet_replica.cc:330] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.050109 5841 raft_consensus.cc:2229] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.050256 5841 raft_consensus.cc:2258] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.050776 5841 ts_tablet_manager.cc:1875] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.053241 5837 ts_tablet_manager.cc:1888] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.053406 5837 log.cc:1196] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f49dd4772f8e46478638b29c009c83ea
I20240507 08:11:58.057176 5840 tablet_bootstrap.cc:654] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.057454 6073 raft_consensus.cc:2726] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.062495 5841 ts_tablet_manager.cc:1888] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.062651 5841 log.cc:1196] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f49dd4772f8e46478638b29c009c83ea
I20240507 08:11:58.063006 5841 ts_tablet_manager.cc:1909] T f49dd4772f8e46478638b29c009c83ea P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.064054 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f49dd4772f8e46478638b29c009c83ea (table default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]) successfully deleted
I20240507 08:11:58.036549 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "191adef34efa4c3999d4642a80177294" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.065040 5837 ts_tablet_manager.cc:1909] T f49dd4772f8e46478638b29c009c83ea P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:11:58.068056 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 already in progress: creating tablet
I20240507 08:11:58.068286 5841 tablet_replica.cc:330] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.068475 5841 raft_consensus.cc:2229] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.068648 5841 raft_consensus.cc:2258] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.069178 5841 ts_tablet_manager.cc:1875] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.070801 6325 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:58.020174 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e3ffaf1781e4b108f305346476905be" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.076341 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f49dd4772f8e46478638b29c009c83ea (table default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]) successfully deleted
W20240507 08:11:58.076172 5517 leader_election.cc:341] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 67c8624285d946e597227430a5714e2f
I20240507 08:11:58.078032 6341 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:58.081005 5841 ts_tablet_manager.cc:1888] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.081130 5841 log.cc:1196] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8ed8bac124cb4de983f0fdbaf728975f
I20240507 08:11:58.085081 6337 raft_consensus.cc:482] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.085328 5841 ts_tablet_manager.cc:1909] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.085248 6337 raft_consensus.cc:504] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.086028 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8ed8bac124cb4de983f0fdbaf728975f (table default.Table_71f59cb7918c42d29419261d5f361830 [id=2e71cf49719f4d9fb07c289e1428080c]) successfully deleted
I20240507 08:11:58.079442 5839 raft_consensus.cc:348] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.087309 5841 tablet_replica.cc:330] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.087533 5841 raft_consensus.cc:2229] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.088374 5841 raft_consensus.cc:2258] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.088615 6337 leader_election.cc:288] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.088948 5841 ts_tablet_manager.cc:1875] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:58.089679 5520 leader_election.cc:341] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:58.095782 6311 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:58.098425 6311 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39de2a69ed4492aab4d7a232b09c1ae, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.099239 6311 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 20, Last appended: 1.20, Last appended by leader: 20, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:58.099695 5517 leader_election.cc:302] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:11:58.100796 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 87913.3us
I20240507 08:11:58.100965 5841 ts_tablet_manager.cc:1888] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.101114 5841 log.cc:1196] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/60bc7bc06e494adcb6a8641b50b90e6c
W20240507 08:11:58.101174 5378 leader_election.cc:341] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.101323 5378 leader_election.cc:302] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.101497 5841 ts_tablet_manager.cc:1909] T 60bc7bc06e494adcb6a8641b50b90e6c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:11:58.101558 5378 raft_consensus.cc:2670] T 8ed8bac124cb4de983f0fdbaf728975f P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:58.102447 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 60bc7bc06e494adcb6a8641b50b90e6c (table default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]) successfully deleted
I20240507 08:11:58.103474 6309 consensus_queue.cc:1035] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:11:58.103925 5378 leader_election.cc:341] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.104112 5378 leader_election.cc:302] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.104532 6309 raft_consensus.cc:2726] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.105892 6337 raft_consensus.cc:2726] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:58.106268 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b27cc38d0c4c4083900cd6765a5cf6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27cc38d0c4c4083900cd6765a5cf6e6 already in progress: creating tablet
W20240507 08:11:58.106745 5378 leader_election.cc:341] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:11:58.106928 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4fcc1d7e309946d783a524b61d5f1269 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fcc1d7e309946d783a524b61d5f1269 already in progress: creating tablet
I20240507 08:11:58.106967 5378 leader_election.cc:302] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.107411 5841 tablet_replica.cc:330] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.107530 6309 raft_consensus.cc:2726] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.107651 5841 raft_consensus.cc:2229] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.108664 5841 raft_consensus.cc:2258] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:11:58.108914 5378 consensus_peers.cc:485] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:58.109092 5839 raft_consensus.cc:374] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.109320 5841 ts_tablet_manager.cc:1875] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.110009 5840 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.110206 5840 ts_tablet_manager.cc:1367] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.084s user 0.005s sys 0.002s
I20240507 08:11:58.110634 5839 raft_consensus.cc:729] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.110906 5839 consensus_queue.cc:260] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.111588 5840 raft_consensus.cc:348] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.111904 5840 raft_consensus.cc:374] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.112042 5840 raft_consensus.cc:729] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.112362 5840 consensus_queue.cc:260] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:58.114651 5517 leader_election.cc:341] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:58.120827 5841 ts_tablet_manager.cc:1888] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.120975 5841 log.cc:1196] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/191adef34efa4c3999d4642a80177294
I20240507 08:11:58.121222 5840 ts_tablet_manager.cc:1398] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240507 08:11:58.121371 5841 ts_tablet_manager.cc:1909] T 191adef34efa4c3999d4642a80177294 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.122375 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 191adef34efa4c3999d4642a80177294 (table default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]) successfully deleted
I20240507 08:11:58.123945 5839 ts_tablet_manager.cc:1398] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.079s user 0.001s sys 0.001s
W20240507 08:11:58.126528 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 47a154769d0f4501837888feba98f33d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47a154769d0f4501837888feba98f33d already in progress: creating tablet
W20240507 08:11:58.127043 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e14487e625374060b118c9cce4500b3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14487e625374060b118c9cce4500b3f already in progress: creating tablet
I20240507 08:11:58.127296 6309 raft_consensus.cc:482] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.127458 6309 raft_consensus.cc:504] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.128343 6309 leader_election.cc:288] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.128968 5840 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:11:58.129793 5377 leader_election.cc:341] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: d9ff5e1b6a9a476c8295a0d0ee85aecc
W20240507 08:11:58.129879 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7d89e4185229431f9a68f599da4aff34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d89e4185229431f9a68f599da4aff34 already in progress: creating tablet
W20240507 08:11:58.130621 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 565ac6091f5e48b1b024094d5560e275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565ac6091f5e48b1b024094d5560e275 already in progress: creating tablet
I20240507 08:11:58.131160 5840 tablet_bootstrap.cc:654] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:58.131410 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 049483b0533642f2b46d8fa0e9b123cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 049483b0533642f2b46d8fa0e9b123cc already in progress: creating tablet
W20240507 08:11:58.131721 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3a94c97ecc5d47abb190f3833fd317ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a94c97ecc5d47abb190f3833fd317ca already in progress: creating tablet
I20240507 08:11:58.137969 5517 leader_election.cc:302] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:58.143416 6073 raft_consensus.cc:2726] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.149046 5838 ts_tablet_manager.cc:1888] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:58.149227 5838 log.cc:1196] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/bc619db3a2944ecc9f0112b28de148a6
I20240507 08:11:58.149507 5839 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:58.152442 5838 ts_tablet_manager.cc:1909] T bc619db3a2944ecc9f0112b28de148a6 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:58.153053 5838 tablet_replica.cc:330] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:58.153905 5838 raft_consensus.cc:2229] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.154162 5838 raft_consensus.cc:2258] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.157246 5840 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.157464 5840 ts_tablet_manager.cc:1367] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.029s user 0.015s sys 0.011s
I20240507 08:11:58.157462 5838 ts_tablet_manager.cc:1875] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.158442 5840 raft_consensus.cc:348] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.158843 5840 raft_consensus.cc:374] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.158936 5840 raft_consensus.cc:729] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.159224 5840 consensus_queue.cc:260] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.161695 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9ff5e1b6a9a476c8295a0d0ee85aecc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:58.166329 5378 leader_election.cc:341] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.166571 5378 leader_election.cc:302] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.167210 6309 raft_consensus.cc:2726] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.169061 5840 ts_tablet_manager.cc:1398] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240507 08:11:58.175266 5839 tablet_bootstrap.cc:654] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.179628 5840 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.182211 5840 tablet_bootstrap.cc:654] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.212858 5840 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.213191 5840 ts_tablet_manager.cc:1367] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.034s user 0.009s sys 0.013s
I20240507 08:11:58.214545 5840 raft_consensus.cc:348] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.214941 5840 raft_consensus.cc:374] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.215088 5840 raft_consensus.cc:729] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.215462 5840 consensus_queue.cc:260] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.217018 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:58.228973 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6828726f0a3b436faf6ac20342edff29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.229564 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6828726f0a3b436faf6ac20342edff29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.234226 5840 ts_tablet_manager.cc:1398] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240507 08:11:58.237603 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet eec3704b135740f1a906d60f3ea87ffe
I20240507 08:11:58.238484 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet eec3704b135740f1a906d60f3ea87ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.238482 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ab27b29d5c594b1cbd6adf0e76b29fa7" } new_table_name: "default.Table_35f061a2f58c4b3fa505b7b2adf6e66d" modify_external_catalogs: true
I20240507 08:11:58.238710 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet eec3704b135740f1a906d60f3ea87ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.239820 6350 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:58.240198 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5405f264bac1464a8f1b4435715a64a4
I20240507 08:11:58.240708 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 5405f264bac1464a8f1b4435715a64a4 deleted and replaced by tablet c20c60fe0aad4cb3b12d04ad1a87d9f4
I20240507 08:11:58.241096 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 5405f264bac1464a8f1b4435715a64a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c20c60fe0aad4cb3b12d04ad1a87d9f4) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.241127 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b2ab50d2ef764827aa64c3a4eaf14190"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:58.245847 5840 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.246421 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet eec3704b135740f1a906d60f3ea87ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.246719 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5405f264bac1464a8f1b4435715a64a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c20c60fe0aad4cb3b12d04ad1a87d9f4) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:58.246850 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet eec3704b135740f1a906d60f3ea87ffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec3704b135740f1a906d60f3ea87ffe already in progress: creating tablet
I20240507 08:11:58.247201 5841 tablet_replica.cc:330] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.247442 5841 raft_consensus.cc:2229] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:58.247795 5841 raft_consensus.cc:2258] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.247970 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 5405f264bac1464a8f1b4435715a64a4 with delete_type TABLET_DATA_DELETED (replaced by tablet c20c60fe0aad4cb3b12d04ad1a87d9f4) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.248226 5837 tablet_replica.cc:330] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:11:58.248412 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2ab50d2ef764827aa64c3a4eaf14190 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.248428 5837 raft_consensus.cc:2229] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.248427 5841 ts_tablet_manager.cc:1875] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.248613 5837 raft_consensus.cc:2258] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.249212 5837 ts_tablet_manager.cc:1875] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:11:58.254289 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet eec3704b135740f1a906d60f3ea87ffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec3704b135740f1a906d60f3ea87ffe already in progress: creating tablet
I20240507 08:11:58.257077 5841 ts_tablet_manager.cc:1888] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:58.257246 5841 log.cc:1196] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5405f264bac1464a8f1b4435715a64a4
I20240507 08:11:58.257673 5841 ts_tablet_manager.cc:1909] T 5405f264bac1464a8f1b4435715a64a4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.257666 5837 ts_tablet_manager.cc:1888] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:58.257797 5837 log.cc:1196] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5405f264bac1464a8f1b4435715a64a4
I20240507 08:11:58.258352 5837 ts_tablet_manager.cc:1909] T 5405f264bac1464a8f1b4435715a64a4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:58.258745 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5405f264bac1464a8f1b4435715a64a4 (table default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]) successfully deleted
I20240507 08:11:58.259475 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5405f264bac1464a8f1b4435715a64a4 (table default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]) successfully deleted
I20240507 08:11:58.261085 5840 tablet_bootstrap.cc:654] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.267202 6073 raft_consensus.cc:482] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.267481 6073 raft_consensus.cc:504] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.268455 6073 leader_election.cc:288] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:58.269992 5517 leader_election.cc:341] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 972bc0a5ba924e0fb946cdacd9575c1e
W20240507 08:11:58.271133 5520 leader_election.cc:341] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 972bc0a5ba924e0fb946cdacd9575c1e
I20240507 08:11:58.271308 5520 leader_election.cc:302] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:58.271346 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:58.272174 6073 raft_consensus.cc:2726] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.272328 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:58.271152 5839 tablet_bootstrap.cc:492] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:58.272948 5839 ts_tablet_manager.cc:1367] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.138s user 0.003s sys 0.004s
I20240507 08:11:58.274330 5839 raft_consensus.cc:348] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.274921 5839 raft_consensus.cc:374] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.275206 5839 raft_consensus.cc:729] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.276319 5839 consensus_queue.cc:260] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.278122 6355 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:11:58.301147 6311 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: configured and running, proceeding with master startup.
W20240507 08:11:58.302755 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 916e33e40ef44d06a359b82d8fe48157 consensus state has no leader
I20240507 08:11:58.304415 5262 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_6e265b95a5364ac98a18cb1a4b2b26f1 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 2 got 0
I20240507 08:11:58.305285 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_225c3dd465f0444a9688b5c17d613ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.306362 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_225c3dd465f0444a9688b5c17d613ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.311317 5838 ts_tablet_manager.cc:1888] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.311520 5838 log.cc:1196] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/cb7acbdb357445adb9ce69103c782a7f
I20240507 08:11:58.311944 5838 ts_tablet_manager.cc:1909] T cb7acbdb357445adb9ce69103c782a7f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:58.312348 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 420) took 1116 ms. Trace:
I20240507 08:11:58.312465 5838 rpcz_store.cc:276] 0507 08:11:57.195759 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.195808 (+ 49us) service_pool.cc:225] Handling call
0507 08:11:58.152979 (+957171us) ts_tablet_manager.cc:1120] Deleting tablet cb7acbdb357445adb9ce69103c782a7f
0507 08:11:58.152988 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.220423 (+ 67435us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:58.311165 (+ 90742us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:58.312278 (+ 1113us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":62,"tablet-delete.queue_time_us":956973}
I20240507 08:11:58.312990 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet cb7acbdb357445adb9ce69103c782a7f (table default.Table_8fd9028ec0ef48ad8c17cbd2f51bd20e [id=f349aa81bb8b4fa286150725527f5f13]) successfully deleted
I20240507 08:11:58.323225 5843 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:58.323432 5843 ts_tablet_manager.cc:1367] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.293s user 0.067s sys 0.099s
I20240507 08:11:58.324205 5843 raft_consensus.cc:348] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.324445 5843 raft_consensus.cc:374] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.324527 5843 raft_consensus.cc:729] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.325435 5843 consensus_queue.cc:260] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.325881 6350 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:58.326481 5843 ts_tablet_manager.cc:1398] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:11:58.326856 5843 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:58.278260 5839 ts_tablet_manager.cc:1398] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:11:58.327394 5839 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:11:58.330128 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 167ee9cbc9174b82824c468e098472a6 consensus state has no leader
I20240507 08:11:58.332486 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d74868790e444b1fb50ea56051f4810c" } modify_external_catalogs: true
I20240507 08:11:58.332724 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d74868790e444b1fb50ea56051f4810c" } modify_external_catalogs: true
I20240507 08:11:58.333302 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_84d691a98c0d402fb6532f0e49f61488" } new_table_name: "default.Table_bf6b95a29b9d466eabf2276567fe5fa9" modify_external_catalogs: true
I20240507 08:11:58.333906 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0357417146364abdb06bb955b43a5fe2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:58.334288 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
W20240507 08:11:58.334492 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0357417146364abdb06bb955b43a5fe2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.335412 5839 tablet_bootstrap.cc:654] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.335867 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 418) took 1144 ms. Trace:
I20240507 08:11:58.335842 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_80a52d858eaf480cab22d552d7c8eff5" } modify_external_catalogs: true
I20240507 08:11:58.336016 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_80a52d858eaf480cab22d552d7c8eff5" } modify_external_catalogs: true
I20240507 08:11:58.335969 5838 rpcz_store.cc:276] 0507 08:11:57.191635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.209645 (+ 18010us) service_pool.cc:225] Handling call
0507 08:11:58.332462 (+1122817us) ts_tablet_manager.cc:1120] Deleting tablet 4e63c3f3ecc84913ad05864be450e9b0
0507 08:11:58.335645 (+ 3183us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.335802 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1122509}
W20240507 08:11:58.336483 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4e63c3f3ecc84913ad05864be450e9b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e63c3f3ecc84913ad05864be450e9b0 already in progress: creating tablet
I20240507 08:11:58.338375 5256 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:58.338717 5256 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 3 got 2
I20240507 08:11:58.340334 5840 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.340497 5838 tablet_replica.cc:330] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:58.340505 5840 ts_tablet_manager.cc:1367] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.095s user 0.019s sys 0.030s
I20240507 08:11:58.341188 5840 raft_consensus.cc:348] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.341413 5840 raft_consensus.cc:374] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.341498 5840 raft_consensus.cc:729] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.341773 5840 consensus_queue.cc:260] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.342262 5840 ts_tablet_manager.cc:1398] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:11:58.344233 5838 raft_consensus.cc:2229] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.344455 5838 raft_consensus.cc:2258] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.345046 5838 ts_tablet_manager.cc:1875] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.352007 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet e4c064b969844d938fddfe7f043c0b92 (table default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277]). Expected version 3 got 2
I20240507 08:11:58.352419 5258 catalog_manager.cc:5484] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } }
I20240507 08:11:58.353042 5840 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.354907 5840 tablet_bootstrap.cc:654] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.356663 5256 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_6e265b95a5364ac98a18cb1a4b2b26f1 [id=42e0c3835e194cd6a54f6ad7d8e4f5ed] alter complete (version 2)
I20240507 08:11:58.360004 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 972bc0a5ba924e0fb946cdacd9575c1e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.367019 6073 raft_consensus.cc:482] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.367244 6073 raft_consensus.cc:504] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.367985 6073 leader_election.cc:288] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.367990 6309 raft_consensus.cc:482] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.368242 6309 raft_consensus.cc:504] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.369117 6309 leader_election.cc:288] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.369412 5560 tablet_service.cc:1467] Processing CreateTablet for tablet b31fdadee60e4f5e94e92c536e18774d (DEFAULT_TABLE table=default.Table_f993482dcb444626a4d006bf1755dcca [id=aa4bf50ad570490b8120f3d75aad746b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.369886 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31fdadee60e4f5e94e92c536e18774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.370970 5683 tablet_service.cc:1467] Processing CreateTablet for tablet b31fdadee60e4f5e94e92c536e18774d (DEFAULT_TABLE table=default.Table_f993482dcb444626a4d006bf1755dcca [id=aa4bf50ad570490b8120f3d75aad746b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.371456 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31fdadee60e4f5e94e92c536e18774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.372951 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:58.373613 5410 tablet_service.cc:1467] Processing CreateTablet for tablet b31fdadee60e4f5e94e92c536e18774d (DEFAULT_TABLE table=default.Table_f993482dcb444626a4d006bf1755dcca [id=aa4bf50ad570490b8120f3d75aad746b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.374020 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b31fdadee60e4f5e94e92c536e18774d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.376689 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0ea74702c4b4514a31f0fad7e7a2f5b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:58.379145 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53409fe13e4a9585dbba648a25cc07" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:11:58.379755 5377 leader_election.cc:341] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.380302 5517 leader_election.cc:302] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:58.381031 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e53409fe13e4a9585dbba648a25cc07" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.381430 6073 raft_consensus.cc:2726] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.382944 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b (DEFAULT_TABLE table=default.Table_f4ad9488db7b43469ea4ec3d4e365ba4 [id=b0b407fea38d4796bb96d178d7c64658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.383356 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43a80d38ed4788a13421b245fe6e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.384752 5686 tablet_service.cc:1467] Processing CreateTablet for tablet d4fac82e12d0437e950d01dbc4117efa (DEFAULT_TABLE table=default.Table_12317138695c42d695226bd546e857ab [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.385141 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fac82e12d0437e950d01dbc4117efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.385694 5425 tablet_service.cc:1467] Processing CreateTablet for tablet d4fac82e12d0437e950d01dbc4117efa (DEFAULT_TABLE table=default.Table_12317138695c42d695226bd546e857ab [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.386183 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fac82e12d0437e950d01dbc4117efa. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.386323 5378 leader_election.cc:341] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.386461 5378 leader_election.cc:302] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.386926 6309 raft_consensus.cc:2726] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.394519 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b (DEFAULT_TABLE table=default.Table_f4ad9488db7b43469ea4ec3d4e365ba4 [id=b0b407fea38d4796bb96d178d7c64658]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:58.395193 5378 consensus_peers.cc:485] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:11:58.395612 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43a80d38ed4788a13421b245fe6e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.397680 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 093fb3986a674b93a435408c7ec85500 (DEFAULT_TABLE table=default.Table_bc5e572b5c614e0fbb526ab59a22254f [id=c72296a9a4f0452fa4ac3809ca07312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.397737 5546 tablet_service.cc:1467] Processing CreateTablet for tablet d4fac82e12d0437e950d01dbc4117efa (DEFAULT_TABLE table=default.Table_12317138695c42d695226bd546e857ab [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.397892 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b (DEFAULT_TABLE table=default.Table_f4ad9488db7b43469ea4ec3d4e365ba4 [id=b0b407fea38d4796bb96d178d7c64658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.398150 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093fb3986a674b93a435408c7ec85500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.398177 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4fac82e12d0437e950d01dbc4117efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.398308 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f43a80d38ed4788a13421b245fe6e8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.403048 5691 tablet_service.cc:1467] Processing CreateTablet for tablet ec08c0d8a1ab444489340415303a5739 (DEFAULT_TABLE table=default.Table_f10e0f30c1f14fa18ead66c614f8473e [id=c4b899bf5b0244ac897da574bda1457d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.403481 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec08c0d8a1ab444489340415303a5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.403836 5413 tablet_service.cc:1467] Processing CreateTablet for tablet ec08c0d8a1ab444489340415303a5739 (DEFAULT_TABLE table=default.Table_f10e0f30c1f14fa18ead66c614f8473e [id=c4b899bf5b0244ac897da574bda1457d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.404222 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec08c0d8a1ab444489340415303a5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.410935 6309 consensus_queue.cc:1035] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:11:58.417022 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 093fb3986a674b93a435408c7ec85500 (DEFAULT_TABLE table=default.Table_bc5e572b5c614e0fbb526ab59a22254f [id=c72296a9a4f0452fa4ac3809ca07312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.417620 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093fb3986a674b93a435408c7ec85500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.424865 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2ed8516944a083c02e07045e44bd (DEFAULT_TABLE table=default.Table_e9691d8ea6744056afe858914bac70c5 [id=fdcccc04f0834c488a1176ec57b02be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.425482 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2ed8516944a083c02e07045e44bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.430438 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 093fb3986a674b93a435408c7ec85500 (DEFAULT_TABLE table=default.Table_bc5e572b5c614e0fbb526ab59a22254f [id=c72296a9a4f0452fa4ac3809ca07312b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.430924 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 093fb3986a674b93a435408c7ec85500. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.434078 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2ed8516944a083c02e07045e44bd (DEFAULT_TABLE table=default.Table_e9691d8ea6744056afe858914bac70c5 [id=fdcccc04f0834c488a1176ec57b02be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.434489 5554 tablet_service.cc:1467] Processing CreateTablet for tablet ec08c0d8a1ab444489340415303a5739 (DEFAULT_TABLE table=default.Table_f10e0f30c1f14fa18ead66c614f8473e [id=c4b899bf5b0244ac897da574bda1457d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.434610 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2ed8516944a083c02e07045e44bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.434959 6203 raft_consensus.cc:482] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.435078 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec08c0d8a1ab444489340415303a5739. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.435119 6203 raft_consensus.cc:504] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.435886 6203 leader_election.cc:288] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:58.436751 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:11:58.437558 5656 leader_election.cc:341] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.439429 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2ed8516944a083c02e07045e44bd (DEFAULT_TABLE table=default.Table_e9691d8ea6744056afe858914bac70c5 [id=fdcccc04f0834c488a1176ec57b02be2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.439530 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.439846 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2ed8516944a083c02e07045e44bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.420070 5843 tablet_bootstrap.cc:654] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:58.441013 5654 leader_election.cc:341] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.441222 5654 leader_election.cc:302] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.441522 6203 raft_consensus.cc:2726] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.444252 5683 tablet_service.cc:1467] Processing CreateTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 (DEFAULT_TABLE table=default.Table_55ca0ba456d0447d95c4f2df7eee0591 [id=1d33bf3079104056bfdd3b875017f483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.444751 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.454833 5555 tablet_service.cc:1467] Processing CreateTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 (DEFAULT_TABLE table=default.Table_55ca0ba456d0447d95c4f2df7eee0591 [id=1d33bf3079104056bfdd3b875017f483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.455351 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.456765 5838 ts_tablet_manager.cc:1888] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.462360 5411 tablet_service.cc:1467] Processing CreateTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 (DEFAULT_TABLE table=default.Table_55ca0ba456d0447d95c4f2df7eee0591 [id=1d33bf3079104056bfdd3b875017f483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.462818 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.463048 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 506fb6f8445a402eb9fbc81c8019cc8b (DEFAULT_TABLE table=default.Table_e0cbf17e47c943ba8fefb68f7b925e07 [id=72da615dacf7461b9aee3e74877f7975]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.463167 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 506fb6f8445a402eb9fbc81c8019cc8b (DEFAULT_TABLE table=default.Table_e0cbf17e47c943ba8fefb68f7b925e07 [id=72da615dacf7461b9aee3e74877f7975]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.463609 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506fb6f8445a402eb9fbc81c8019cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.464135 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 506fb6f8445a402eb9fbc81c8019cc8b (DEFAULT_TABLE table=default.Table_e0cbf17e47c943ba8fefb68f7b925e07 [id=72da615dacf7461b9aee3e74877f7975]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.464516 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506fb6f8445a402eb9fbc81c8019cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.468884 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506fb6f8445a402eb9fbc81c8019cc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.474133 5838 log.cc:1196] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/ea17889619fa420fbeacf1b41594c598
I20240507 08:11:58.476768 5843 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:58.476929 5843 ts_tablet_manager.cc:1367] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.150s user 0.002s sys 0.002s
I20240507 08:11:58.494532 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 01a9165fdc41482cb1f8431ab6479126 (DEFAULT_TABLE table=default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f [id=825a7b456d14442ab083c2700bed6a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.495034 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a9165fdc41482cb1f8431ab6479126. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.497814 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 077ccc3946b34008bea7b80d8aee2662 consensus state has no leader
I20240507 08:11:58.500137 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bc44d932cc29477e8751f038712ef26c" } new_table_name: "default.Table_5b668756389c48ef943681187557167a" modify_external_catalogs: true
I20240507 08:11:58.501741 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_996bd171383346f6b81bdcfc316f9ee4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.502314 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_996bd171383346f6b81bdcfc316f9ee4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.503221 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2c361f4a301a46c98bcbb99d728d535f" } new_table_name: "default.Table_90ae737e5e994b0bb0be30526093e6de" modify_external_catalogs: true
I20240507 08:11:58.505563 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 01a9165fdc41482cb1f8431ab6479126 (DEFAULT_TABLE table=default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f [id=825a7b456d14442ab083c2700bed6a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.506035 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a9165fdc41482cb1f8431ab6479126. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.512846 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 167ee9cbc9174b82824c468e098472a6 (table default.Table_35f061a2f58c4b3fa505b7b2adf6e66d [id=c5ee2823c21745f380337f43009224f7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 167ee9cbc9174b82824c468e098472a6 consensus state has no leader
I20240507 08:11:58.504935 5843 raft_consensus.cc:348] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.513422 5843 raft_consensus.cc:374] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.514770 5843 raft_consensus.cc:729] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.520639 5843 consensus_queue.cc:260] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:58.523926 5258 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.193:55019 (request call id 75) took 809 ms (client timeout 1000 ms). Trace:
W20240507 08:11:58.524086 5258 rpcz_store.cc:269] 0507 08:11:57.714901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.715958 (+ 1057us) service_pool.cc:225] Handling call
0507 08:11:58.523878 (+807920us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:11:58.386038 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:11:58.386155 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:11:58.386163 (+ 8us) write_op.cc:435] Acquired schema lock
0507 08:11:58.386165 (+ 2us) tablet.cc:660] Decoding operations
0507 08:11:58.386285 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:11:58.386299 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0507 08:11:58.386303 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:11:58.386370 (+ 67us) tablet.cc:699] Row locks acquired
0507 08:11:58.386372 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:11:58.386418 (+ 46us) write_op.cc:270] Start()
0507 08:11:58.386455 (+ 37us) write_op.cc:276] Timestamp: P: 1715069518386367 usec, L: 0
0507 08:11:58.386458 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:11:58.500167 (+113709us) log.cc:842] Serialized 533 byte log entry
0507 08:11:58.504207 (+ 4040us) spinlock_profiling.cc:243] Waited 113 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 000000000095519a 00007f3bc4c1f162 00007f3bc5f87e32 00007f3bc5f86959 00007f3bc5f8e86b 00007f3bc5f8e6e8 0000000000a60bcc 00007f3bb9420b74 00007f3bb942419b 00007f3bb9424018 0000000000a60bcc
0507 08:11:58.521706 (+ 17499us) op_driver.cc:444] REPLICATION: finished
0507 08:11:58.522188 (+ 482us) write_op.cc:301] APPLY: starting
0507 08:11:58.522204 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0507 08:11:58.522240 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0507 08:11:58.522242 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:11:58.522284 (+ 42us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:11:58.522321 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:11:58.522328 (+ 7us) write_op.cc:312] APPLY: finished
0507 08:11:58.522393 (+ 65us) log.cc:842] Serialized 23 byte log entry
0507 08:11:58.522453 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:11:58.522468 (+ 15us) write_op.cc:454] Released schema lock
0507 08:11:58.522474 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0507 08:11:58.522530 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:11:58.522535 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"child_traces":[["op",{"apply.queue_time_us":438,"apply.run_cpu_time_us":391,"apply.run_wall_time_us":391,"num_ops":1,"prepare.queue_time_us":32623,"prepare.run_cpu_time_us":2297,"prepare.run_wall_time_us":118273,"raft.queue_time_us":10953,"raft.run_cpu_time_us":229,"raft.run_wall_time_us":229,"replication_time_us":135226,"spinlock_wait_cycles":261004160}]]}
I20240507 08:11:58.526264 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 53fb38b567c5455ebc441e5f71db694f
I20240507 08:11:58.527070 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_287233b419e646b58b404eb7190a3ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.527621 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_287233b419e646b58b404eb7190a3ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.528846 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5e0fb75933874a948281bf573a1723a1
I20240507 08:11:58.529374 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 31251576beee4a8a81a453b721c9c058 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:58.529852 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 972bc0a5ba924e0fb946cdacd9575c1e from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:58.530201 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet f5e48a2b4a0e4088ad009a7f8fe77f3f from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:58.530573 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 167ee9cbc9174b82824c468e098472a6 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:58.532393 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 63ccdad494794192b3c37c8d5cbfcb17
I20240507 08:11:58.533161 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5b058aadbe304a9b9c8ffd023c1a0470"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:58.535452 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0fb75933874a948281bf573a1723a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.536497 5841 tablet_replica.cc:330] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.536458 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:11:58.536710 5841 raft_consensus.cc:2229] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.536877 5841 raft_consensus.cc:2258] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.537508 5841 ts_tablet_manager.cc:1875] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.538431 5255 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:58.539078 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:11:58.539804 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4cf5e458eb7e4fe4b1a953b18d982297"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.540313 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cf5e458eb7e4fe4b1a953b18d982297 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:58.543709 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53bc9bf1c04e4015a152cb750184c9ce consensus state has no leader
I20240507 08:11:58.544039 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9342125a817c439eb044fbe82437c21a" } modify_external_catalogs: true
I20240507 08:11:58.544209 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9342125a817c439eb044fbe82437c21a" } modify_external_catalogs: true
I20240507 08:11:58.544946 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 55a535a1039d43b9b0bf2c61304c2d01 (DEFAULT_TABLE table=default.Table_336e8bfc36dd4d50a3a52a778ef1f102 [id=060d408d159e4c21877717b9f743d93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.545428 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a535a1039d43b9b0bf2c61304c2d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.548930 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.548955 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:58.549710 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b058aadbe304a9b9c8ffd023c1a0470 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.554215 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.554389 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0fb75933874a948281bf573a1723a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.554528 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.554869 5837 tablet_replica.cc:330] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:58.555078 5837 raft_consensus.cc:2229] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.555229 5837 raft_consensus.cc:2258] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.556248 5837 ts_tablet_manager.cc:1875] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.556283 5838 ts_tablet_manager.cc:1909] T ea17889619fa420fbeacf1b41594c598 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:58.556605 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:58.560489 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 63ccdad494794192b3c37c8d5cbfcb17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63ccdad494794192b3c37c8d5cbfcb17 already in progress: creating tablet
W20240507 08:11:58.560772 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5e0fb75933874a948281bf573a1723a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e0fb75933874a948281bf573a1723a1 already in progress: creating tablet
I20240507 08:11:58.561298 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.561545 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fb38b567c5455ebc441e5f71db694f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.561851 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 422) took 1336 ms. Trace:
I20240507 08:11:58.562039 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 01a9165fdc41482cb1f8431ab6479126 (DEFAULT_TABLE table=default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f [id=825a7b456d14442ab083c2700bed6a98]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.562908 5838 rpcz_store.cc:276] 0507 08:11:57.223420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.223470 (+ 50us) service_pool.cc:225] Handling call
0507 08:11:58.340405 (+1116935us) ts_tablet_manager.cc:1120] Deleting tablet ea17889619fa420fbeacf1b41594c598
0507 08:11:58.340415 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.383415 (+ 43000us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:58.441144 (+ 57729us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:58.560073 (+118929us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1116703}
I20240507 08:11:58.563607 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet ea17889619fa420fbeacf1b41594c598 (table default.Table_2f5246f5a6874c848d060c15dd9be10f [id=f5c842b1870e408799094038bc1b1afa]) successfully deleted
I20240507 08:11:58.563925 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0fb75933874a948281bf573a1723a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.564733 5837 ts_tablet_manager.cc:1888] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.565014 5837 log.cc:1196] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:11:58.565408 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 421) took 1342 ms. Trace:
I20240507 08:11:58.565604 5837 ts_tablet_manager.cc:1909] T c84a2018f1c442e2a4dfc1c6b9c5ceef P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:58.566092 5838 rpcz_store.cc:276] 0507 08:11:57.223303 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.225185 (+ 1882us) service_pool.cc:225] Handling call
0507 08:11:58.565269 (+1340084us) ts_tablet_manager.cc:1120] Deleting tablet 7841f5ebf5ed4f6eb28dcbd4087f1713
0507 08:11:58.565278 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.565375 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1339848}
W20240507 08:11:58.566524 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 already in progress: creating tablet
I20240507 08:11:58.566880 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 424) took 1257 ms. Trace:
W20240507 08:11:58.568729 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6f8f4205f1f48abaf4fcd7113ad44a8 consensus state has no leader
I20240507 08:11:58.569176 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4fa2d0d8a1a9482599a4d430029b2b4b" } new_table_name: "default.Table_a48b5e1d22504c999c0df606f5092147" modify_external_catalogs: true
I20240507 08:11:58.570096 5838 rpcz_store.cc:276] 0507 08:11:57.309520 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.309609 (+ 89us) service_pool.cc:225] Handling call
0507 08:11:58.566757 (+1257148us) ts_tablet_manager.cc:1120] Deleting tablet 4fcc1d7e309946d783a524b61d5f1269
0507 08:11:58.566766 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.566851 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1256907}
I20240507 08:11:58.570858 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_139fa8ac0afe4140ac71e91821a98481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:58.571272 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8c0ea847a8d142b3a28ac68403f103ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.571460 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_139fa8ac0afe4140ac71e91821a98481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:58.571760 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c0ea847a8d142b3a28ac68403f103ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:58.572805 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4fcc1d7e309946d783a524b61d5f1269 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fcc1d7e309946d783a524b61d5f1269 already in progress: creating tablet
I20240507 08:11:58.575337 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 425) took 1265 ms. Trace:
I20240507 08:11:58.575753 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (table default.Table_b1add7221be940138ee5b9e2e796e92c [id=70d5dddc40a3453bb64b8019c2a18046]) successfully deleted
I20240507 08:11:58.576193 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fb38b567c5455ebc441e5f71db694f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.576411 5838 rpcz_store.cc:276] 0507 08:11:57.309671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.309896 (+ 225us) service_pool.cc:225] Handling call
0507 08:11:58.575184 (+1265288us) ts_tablet_manager.cc:1120] Deleting tablet b27cc38d0c4c4083900cd6765a5cf6e6
0507 08:11:58.575194 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:58.575295 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1265126}
I20240507 08:11:58.576833 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:58.577023 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b27cc38d0c4c4083900cd6765a5cf6e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27cc38d0c4c4083900cd6765a5cf6e6 already in progress: creating tablet
I20240507 08:11:58.577169 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fb38b567c5455ebc441e5f71db694f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.577558 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.578097 5843 ts_tablet_manager.cc:1398] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.101s user 0.002s sys 0.000s
I20240507 08:11:58.579083 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01a9165fdc41482cb1f8431ab6479126. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.584748 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:58.585709 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:11:58.585966 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:11:58.587308 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 55a535a1039d43b9b0bf2c61304c2d01 (DEFAULT_TABLE table=default.Table_336e8bfc36dd4d50a3a52a778ef1f102 [id=060d408d159e4c21877717b9f743d93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.588793 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_e5036f51d41048c1bd6c65fbb8298939 [id=2435a4b9c9a7464aafa19339bf7c9277] alter complete (version 3)
I20240507 08:11:58.589138 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a535a1039d43b9b0bf2c61304c2d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.589937 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_483c951eac554e4c9a5a5f6e16e2b109"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.590507 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_483c951eac554e4c9a5a5f6e16e2b109 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.590570 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6e265b95a5364ac98a18cb1a4b2b26f1" } new_table_name: "default.Table_e85ccc16434742ac812466c315098b9a" modify_external_catalogs: true
I20240507 08:11:58.592263 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 167ee9cbc9174b82824c468e098472a6
I20240507 08:11:58.592336 5843 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:58.592759 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 167ee9cbc9174b82824c468e098472a6 deleted and replaced by tablet 6563c42d240b4e8685dfa4972f2b9924
I20240507 08:11:58.593111 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9685ac4fffc44198ac025a3ca19e81dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.593624 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9685ac4fffc44198ac025a3ca19e81dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.595376 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 167ee9cbc9174b82824c468e098472a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6563c42d240b4e8685dfa4972f2b9924) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.580333 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 55a535a1039d43b9b0bf2c61304c2d01 (DEFAULT_TABLE table=default.Table_336e8bfc36dd4d50a3a52a778ef1f102 [id=060d408d159e4c21877717b9f743d93d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.595829 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb (DEFAULT_TABLE table=default.Table_6f159b4cb58147b884dd65d03fe08c54 [id=4c76da1d998b4e41a651d7cd200ad100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.596382 5416 tablet_service.cc:1467] Processing CreateTablet for tablet f513573e340547d6ae5ab848d00ace72 (DEFAULT_TABLE table=default.Table_825e44ca78c14d6392c7e6864babde4c [id=a2ce6544eeb7467cb6de0ecccd4ab6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.597012 5411 tablet_service.cc:1467] Processing CreateTablet for tablet f64e338784704312b986762a143dda4c (DEFAULT_TABLE table=default.Table_cd8f372c67b74794a5ac56adc1990bad [id=c7918763c3e948c1a997275ef2fa4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.598254 5843 tablet_bootstrap.cc:654] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.599391 5699 tablet_service.cc:1467] Processing CreateTablet for tablet f64e338784704312b986762a143dda4c (DEFAULT_TABLE table=default.Table_cd8f372c67b74794a5ac56adc1990bad [id=c7918763c3e948c1a997275ef2fa4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.599844 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64e338784704312b986762a143dda4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.600375 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 190baa48859640cf928642dccbebfceb (DEFAULT_TABLE table=default.Table_361a1e87a0ae409dbfa844e2b985885e [id=fea5bea11b1049f6952bad1e4404caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.600471 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet 167ee9cbc9174b82824c468e098472a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6563c42d240b4e8685dfa4972f2b9924) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.601316 5558 tablet_service.cc:1467] Processing CreateTablet for tablet f64e338784704312b986762a143dda4c (DEFAULT_TABLE table=default.Table_cd8f372c67b74794a5ac56adc1990bad [id=c7918763c3e948c1a997275ef2fa4dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.601719 5410 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3397e33b4cc7805a25e7535a383a (DEFAULT_TABLE table=default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa [id=26980c9347ea407ab58f10f39b2657fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.601862 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64e338784704312b986762a143dda4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.604167 5840 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.604322 5840 ts_tablet_manager.cc:1367] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.251s user 0.031s sys 0.061s
I20240507 08:11:58.604543 5562 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3397e33b4cc7805a25e7535a383a (DEFAULT_TABLE table=default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa [id=26980c9347ea407ab58f10f39b2657fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.604678 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb (DEFAULT_TABLE table=default.Table_6f159b4cb58147b884dd65d03fe08c54 [id=4c76da1d998b4e41a651d7cd200ad100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.604938 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3397e33b4cc7805a25e7535a383a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.605439 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 190baa48859640cf928642dccbebfceb (DEFAULT_TABLE table=default.Table_361a1e87a0ae409dbfa844e2b985885e [id=fea5bea11b1049f6952bad1e4404caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.606002 5554 tablet_service.cc:1467] Processing CreateTablet for tablet f513573e340547d6ae5ab848d00ace72 (DEFAULT_TABLE table=default.Table_825e44ca78c14d6392c7e6864babde4c [id=a2ce6544eeb7467cb6de0ecccd4ab6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.606340 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005100f64fd49a3ac29d34aa728fdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.607195 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190baa48859640cf928642dccbebfceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.609663 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190baa48859640cf928642dccbebfceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.611243 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005100f64fd49a3ac29d34aa728fdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.611508 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55a535a1039d43b9b0bf2c61304c2d01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.611816 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f513573e340547d6ae5ab848d00ace72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.612210 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3397e33b4cc7805a25e7535a383a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.612690 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 53fb38b567c5455ebc441e5f71db694f because the tablet was not found. No further retry: Not found: Tablet not found: 53fb38b567c5455ebc441e5f71db694f
I20240507 08:11:58.616206 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb (DEFAULT_TABLE table=default.Table_6f159b4cb58147b884dd65d03fe08c54 [id=4c76da1d998b4e41a651d7cd200ad100]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.616433 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 190baa48859640cf928642dccbebfceb (DEFAULT_TABLE table=default.Table_361a1e87a0ae409dbfa844e2b985885e [id=fea5bea11b1049f6952bad1e4404caa0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.616485 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 167ee9cbc9174b82824c468e098472a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6563c42d240b4e8685dfa4972f2b9924) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.616670 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8005100f64fd49a3ac29d34aa728fdfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.616935 5696 tablet_service.cc:1467] Processing CreateTablet for tablet b6ba3397e33b4cc7805a25e7535a383a (DEFAULT_TABLE table=default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa [id=26980c9347ea407ab58f10f39b2657fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.617285 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6ba3397e33b4cc7805a25e7535a383a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.617939 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 167ee9cbc9174b82824c468e098472a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167ee9cbc9174b82824c468e098472a6 already in progress: creating tablet
I20240507 08:11:58.618563 5840 raft_consensus.cc:348] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.618808 5840 raft_consensus.cc:374] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.618817 5681 tablet_service.cc:1467] Processing CreateTablet for tablet fccc127d380d4035905c1b152ab1017b (DEFAULT_TABLE table=default.Table_b9fe51f8eb5e42aaa715ab338658f664 [id=0f3cf1f48e2b460eacb9bacfa824dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.616212 5683 tablet_service.cc:1467] Processing CreateTablet for tablet f513573e340547d6ae5ab848d00ace72 (DEFAULT_TABLE table=default.Table_825e44ca78c14d6392c7e6864babde4c [id=a2ce6544eeb7467cb6de0ecccd4ab6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.621435 5840 raft_consensus.cc:729] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.621763 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190baa48859640cf928642dccbebfceb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.625653 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccc127d380d4035905c1b152ab1017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.626353 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a (DEFAULT_TABLE table=default.Table_7c7240e1d89c4c05a124c7126954e1ae [id=5daa69678d6c43ee9090fcc2242687e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.626950 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b (DEFAULT_TABLE table=default.Table_aa9ec40b8ef74e34994df20db4800454 [id=321276b010e745d59b4ea25439c3e0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.627211 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 53fb38b567c5455ebc441e5f71db694f (DEFAULT_TABLE table=default.Table_d74868790e444b1fb50ea56051f4810c [id=a0107471377c4e6289c8c1871acbccd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.628203 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a (DEFAULT_TABLE table=default.Table_7c7240e1d89c4c05a124c7126954e1ae [id=5daa69678d6c43ee9090fcc2242687e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.628376 5840 consensus_queue.cc:260] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.628919 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4 (DEFAULT_TABLE table=default.Table_115a2eafb15744289ae668ac98becc7c [id=32ae329d138542a785248e171c80ea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.629902 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4e5439844f384513b4719cfe4f4e5d9b
W20240507 08:11:58.630276 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fce98f472ef46589dbea6b6aefa6572 consensus state has no leader
I20240507 08:11:58.631140 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dac7a1a368f344b9808ba5d8a776b1c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.631664 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dac7a1a368f344b9808ba5d8a776b1c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.632309 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a (DEFAULT_TABLE table=default.Table_479f84438e6548f1806133a52949d796 [id=9ce6d0a3a88f4a42952bc1413d286ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.632726 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 53fb38b567c5455ebc441e5f71db694f (DEFAULT_TABLE table=default.Table_d74868790e444b1fb50ea56051f4810c [id=a0107471377c4e6289c8c1871acbccd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.633149 5423 tablet_service.cc:1467] Processing CreateTablet for tablet fccc127d380d4035905c1b152ab1017b (DEFAULT_TABLE table=default.Table_b9fe51f8eb5e42aaa715ab338658f664 [id=0f3cf1f48e2b460eacb9bacfa824dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.633711 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a (DEFAULT_TABLE table=default.Table_479f84438e6548f1806133a52949d796 [id=9ce6d0a3a88f4a42952bc1413d286ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.634065 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f64e338784704312b986762a143dda4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.634622 5420 tablet_service.cc:1467] Processing CreateTablet for tablet aa806508ec3e489a9b2c2fa9196198d5 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.634894 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 5c49745fce2949d1909baf7583b157a7 (DEFAULT_TABLE table=default.Table_14fe89e6f91f4d1593dde48abd8bebda [id=b0b4a6c29284434fad808b6766f477a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.634826 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 (DEFAULT_TABLE table=default.Table_a5fe82b3814f4ba4be0ad9369bbff3e8 [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.637176 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 5c49745fce2949d1909baf7583b157a7 (DEFAULT_TABLE table=default.Table_14fe89e6f91f4d1593dde48abd8bebda [id=b0b4a6c29284434fad808b6766f477a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.640178 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.631103 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1a05b4d9237c454db3fb125ff4f6291b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.640859 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a05b4d9237c454db3fb125ff4f6291b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.647301 5553 tablet_service.cc:1467] Processing CreateTablet for tablet aa806508ec3e489a9b2c2fa9196198d5 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.647655 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f513573e340547d6ae5ab848d00ace72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.649024 5551 tablet_service.cc:1467] Processing CreateTablet for tablet fccc127d380d4035905c1b152ab1017b (DEFAULT_TABLE table=default.Table_b9fe51f8eb5e42aaa715ab338658f664 [id=0f3cf1f48e2b460eacb9bacfa824dc58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.649538 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccc127d380d4035905c1b152ab1017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.650171 5689 tablet_service.cc:1467] Processing CreateTablet for tablet aa806508ec3e489a9b2c2fa9196198d5 (DEFAULT_TABLE table=default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.650719 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f513573e340547d6ae5ab848d00ace72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.650843 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b (DEFAULT_TABLE table=default.Table_aa9ec40b8ef74e34994df20db4800454 [id=321276b010e745d59b4ea25439c3e0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.651099 5840 ts_tablet_manager.cc:1398] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.025s user 0.001s sys 0.002s
I20240507 08:11:58.651422 5840 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.652047 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddb7dd28f43488581d3a75bf1d68a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.652312 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 (DEFAULT_TABLE table=default.Table_a5fe82b3814f4ba4be0ad9369bbff3e8 [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.652470 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa806508ec3e489a9b2c2fa9196198d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.652714 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.652993 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.653275 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3769eb670ff442e2ad2fa5b6a934512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.653820 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.654629 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fb38b567c5455ebc441e5f71db694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.655294 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c49745fce2949d1909baf7583b157a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.655790 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4e5439844f384513b4719cfe4f4e5d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5439844f384513b4719cfe4f4e5d9b already in progress: creating tablet
I20240507 08:11:58.657534 5838 tablet_replica.cc:330] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:58.660867 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4 (DEFAULT_TABLE table=default.Table_115a2eafb15744289ae668ac98becc7c [id=32ae329d138542a785248e171c80ea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.663838 5840 tablet_bootstrap.cc:654] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.663904 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.674887 5843 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:58.675099 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_0856317e9911474aa5861773d7185325 [id=0a1bc06a351549d5aea075ad8d026476] alter complete (version 2)
I20240507 08:11:58.675134 5843 ts_tablet_manager.cc:1367] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.000s
I20240507 08:11:58.675987 5843 raft_consensus.cc:348] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.676991 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.677340 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.677397 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa806508ec3e489a9b2c2fa9196198d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.677664 5843 raft_consensus.cc:374] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.677796 5843 raft_consensus.cc:729] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.678164 5843 consensus_queue.cc:260] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.681283 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a (DEFAULT_TABLE table=default.Table_479f84438e6548f1806133a52949d796 [id=9ce6d0a3a88f4a42952bc1413d286ac5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.681540 5841 ts_tablet_manager.cc:1888] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.681689 5841 log.cc:1196] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5e0fb75933874a948281bf573a1723a1
I20240507 08:11:58.681648 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 (DEFAULT_TABLE table=default.Table_a5fe82b3814f4ba4be0ad9369bbff3e8 [id=0b044cc019d845b9a033666c78e88492]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.682058 5841 ts_tablet_manager.cc:1909] T 5e0fb75933874a948281bf573a1723a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.682066 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.683679 5840 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.683874 5840 ts_tablet_manager.cc:1367] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.004s
I20240507 08:11:58.684373 6203 raft_consensus.cc:482] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.684535 6203 raft_consensus.cc:504] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.684747 5840 raft_consensus.cc:348] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.684973 5840 raft_consensus.cc:374] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.685040 5840 raft_consensus.cc:729] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.684998 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 5c49745fce2949d1909baf7583b157a7 (DEFAULT_TABLE table=default.Table_14fe89e6f91f4d1593dde48abd8bebda [id=b0b4a6c29284434fad808b6766f477a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.685287 5840 consensus_queue.cc:260] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.684623 5838 raft_consensus.cc:2229] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.686494 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddb7dd28f43488581d3a75bf1d68a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.689000 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c49745fce2949d1909baf7583b157a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.690682 5843 ts_tablet_manager.cc:1398] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240507 08:11:58.690873 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b (DEFAULT_TABLE table=default.Table_aa9ec40b8ef74e34994df20db4800454 [id=321276b010e745d59b4ea25439c3e0ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.690214 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 53fb38b567c5455ebc441e5f71db694f (DEFAULT_TABLE table=default.Table_d74868790e444b1fb50ea56051f4810c [id=a0107471377c4e6289c8c1871acbccd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.691246 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a (DEFAULT_TABLE table=default.Table_7c7240e1d89c4c05a124c7126954e1ae [id=5daa69678d6c43ee9090fcc2242687e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.692672 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3769eb670ff442e2ad2fa5b6a934512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.693197 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a397c40278141b7bb80ea5fcdbb125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.693645 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fb38b567c5455ebc441e5f71db694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.694281 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fccc127d380d4035905c1b152ab1017b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.694730 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ddb7dd28f43488581d3a75bf1d68a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.695065 5843 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:58.696388 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.696694 5838 raft_consensus.cc:2258] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.697763 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 59dbc2eaa45d4da58951b45423f284fd (DEFAULT_TABLE table=default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.698661 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 59dbc2eaa45d4da58951b45423f284fd (DEFAULT_TABLE table=default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.695173 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa806508ec3e489a9b2c2fa9196198d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.699661 5688 tablet_service.cc:1467] Processing CreateTablet for tablet aa485207d0f140c99a9386cb4002c480 (DEFAULT_TABLE table=default.Table_6a2e33773a684a20903120d0e1c2e4c3 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.699999 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dbc2eaa45d4da58951b45423f284fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.701654 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a397c40278141b7bb80ea5fcdbb125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.702149 6203 leader_election.cc:288] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:58.703176 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff (DEFAULT_TABLE table=default.Table_c1becaf1c2694657b87263a7e941b119 [id=352055be23d04485b3896bd169ea07ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.703900 5561 tablet_service.cc:1467] Processing CreateTablet for tablet aa485207d0f140c99a9386cb4002c480 (DEFAULT_TABLE table=default.Table_6a2e33773a684a20903120d0e1c2e4c3 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.704233 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4 (DEFAULT_TABLE table=default.Table_115a2eafb15744289ae668ac98becc7c [id=32ae329d138542a785248e171c80ea97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.704540 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a397c40278141b7bb80ea5fcdbb125b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.707427 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.710137 5656 leader_election.cc:341] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 60bc7bc06e494adcb6a8641b50b90e6c
I20240507 08:11:58.710602 5837 tablet_replica.cc:330] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:58.710824 5837 raft_consensus.cc:2229] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.711012 5837 raft_consensus.cc:2258] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.711567 5837 ts_tablet_manager.cc:1875] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.714206 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_291925856aa04048bb7d289bb6837ec0" } new_table_name: "default.Table_8a58f4e04c4147368d43d162c9613c9f" modify_external_catalogs: true
W20240507 08:11:58.717315 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef because the tablet was not found. No further retry: Not found: Tablet not found: c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:11:58.718215 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c49745fce2949d1909baf7583b157a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.725685 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.730191 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279e95d9804c44eaa2c23a4ec02c84ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.719223 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.733525 5584 raft_consensus.cc:3037] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:58.733978 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 59dbc2eaa45d4da58951b45423f284fd (DEFAULT_TABLE table=default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.735239 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.738288 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5e0fb75933874a948281bf573a1723a1 (table default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]) successfully deleted
I20240507 08:11:58.739455 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:58.741206 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 63ccdad494794192b3c37c8d5cbfcb17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63ccdad494794192b3c37c8d5cbfcb17 already in progress: creating tablet
I20240507 08:11:58.741304 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.743229 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279e95d9804c44eaa2c23a4ec02c84ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.748523 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa485207d0f140c99a9386cb4002c480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.750188 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.758214 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c6bf901b2c4a491e8aa4c357670a5d08" } modify_external_catalogs: true
I20240507 08:11:58.758455 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c6bf901b2c4a491e8aa4c357670a5d08" } modify_external_catalogs: true
I20240507 08:11:58.760716 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6" } modify_external_catalogs: true
I20240507 08:11:58.760736 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5b58be7978724a40b7c1020659a68b1c" } new_table_name: "default.Table_a2b806f32a0d49a69d6eb9eb5f235f52" modify_external_catalogs: true
I20240507 08:11:58.760943 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6" } modify_external_catalogs: true
I20240507 08:11:58.762387 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c03c02bf7c44286b5e1273ff2b738c0" } modify_external_catalogs: true
I20240507 08:11:58.762657 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c03c02bf7c44286b5e1273ff2b738c0" } modify_external_catalogs: true
I20240507 08:11:58.764506 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dbc2eaa45d4da58951b45423f284fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.767391 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_46b030f735704339a1921f803d7af17d" } modify_external_catalogs: true
I20240507 08:11:58.767632 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_46b030f735704339a1921f803d7af17d" } modify_external_catalogs: true
I20240507 08:11:58.771502 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e5036f51d41048c1bd6c65fbb8298939" } new_table_name: "default.Table_2071d1f910a644788fb77006559bb1eb" modify_external_catalogs: true
I20240507 08:11:58.801787 6378 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:58.812884 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_22d1db9ba344416091aab96381212e34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.813668 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22d1db9ba344416091aab96381212e34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:58.829690 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28e5a380f2784e3a806afcbb5cd2519a consensus state has no leader
I20240507 08:11:58.830531 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 072702ef2a6d43248132da6aa3a2d493
I20240507 08:11:58.831386 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:11:58.832051 5264 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 5a9a71a16ef94c86872721928ecd934a
I20240507 08:11:58.832273 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0d617213b959452ab6519809e56db128
W20240507 08:11:58.834127 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c4cd7b56d4e468eb754d9653117d023 consensus state has no leader
I20240507 08:11:58.838457 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2e61893054c7404abceb0fc5bc3b08dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.839133 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e61893054c7404abceb0fc5bc3b08dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.844535 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f38a29b591904253b1165f8512dce2a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.845228 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f38a29b591904253b1165f8512dce2a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.848760 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2ee2e89703be4de1a6f05d13e2af9abb" } new_table_name: "default.Table_f7d324ad4a784569bdf917a46203715f" modify_external_catalogs: true
I20240507 08:11:58.850174 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_361a1e87a0ae409dbfa844e2b985885e" } modify_external_catalogs: true
I20240507 08:11:58.850358 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_361a1e87a0ae409dbfa844e2b985885e" } modify_external_catalogs: true
I20240507 08:11:58.852553 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.853947 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.857481 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.858597 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:58.860018 5837 ts_tablet_manager.cc:1888] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.860203 5837 log.cc:1196] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:11:58.860661 5837 ts_tablet_manager.cc:1909] T f5e48a2b4a0e4088ad009a7f8fe77f3f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:58.861877 5837 tablet_replica.cc:330] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:58.862143 5837 raft_consensus.cc:2229] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.862396 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:11:58.863025 5837 raft_consensus.cc:2258] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.864495 5837 ts_tablet_manager.cc:1875] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.864766 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:58.871381 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f0b399a2d074f4ca3626ff17bd5518e consensus state has no leader
I20240507 08:11:58.872239 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_746e0a45757a4d1e898c096f11542589"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.872828 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_746e0a45757a4d1e898c096f11542589 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.877529 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0ae24aad0d2044a98f1f464c8a31d715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.878298 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ae24aad0d2044a98f1f464c8a31d715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.756007 5841 tablet_replica.cc:330] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.816980 5418 tablet_service.cc:1467] Processing CreateTablet for tablet aa485207d0f140c99a9386cb4002c480 (DEFAULT_TABLE table=default.Table_6a2e33773a684a20903120d0e1c2e4c3 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:58.837754 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.884753 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.837945 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:58.845726 6240 raft_consensus.cc:482] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.886031 6240 raft_consensus.cc:504] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.886775 6240 leader_election.cc:288] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.888808 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:58.889099 5718 raft_consensus.cc:2454] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:58.889504 5377 leader_election.cc:302] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:58.889808 6240 raft_consensus.cc:2781] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:58.889916 6240 raft_consensus.cc:482] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:58.889992 6240 raft_consensus.cc:3037] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:58.891813 5841 raft_consensus.cc:2229] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.892043 5841 raft_consensus.cc:2258] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.892791 6240 raft_consensus.cc:504] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.893483 5841 ts_tablet_manager.cc:1875] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.894110 6240 leader_election.cc:288] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.859992 6377 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:58.894425 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:58.894651 5718 raft_consensus.cc:3037] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:58.898458 5718 raft_consensus.cc:2454] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:58.898844 5377 leader_election.cc:302] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:58.899196 6240 raft_consensus.cc:2781] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:58.899359 6240 raft_consensus.cc:686] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:58.899646 6240 consensus_queue.cc:237] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.899775 5838 ts_tablet_manager.cc:1875] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.736785 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60bc7bc06e494adcb6a8641b50b90e6c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.900245 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa485207d0f140c99a9386cb4002c480. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.901786 5654 leader_election.cc:341] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.901839 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa485207d0f140c99a9386cb4002c480. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.901937 5654 leader_election.cc:302] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:11:58.902150 5654 raft_consensus.cc:2670] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:11:58.902479 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.902791 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:11:58.903718 5378 leader_election.cc:341] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.904238 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:58.907620 5840 ts_tablet_manager.cc:1398] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.224s user 0.028s sys 0.049s
I20240507 08:11:58.910393 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:58.917953 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_761e0fb1290f435f82ab49c01e433ad3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:58.918599 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_761e0fb1290f435f82ab49c01e433ad3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:58.918819 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0856317e9911474aa5861773d7185325" } new_table_name: "default.Table_9735652cdc0d447080df766780402da6" modify_external_catalogs: true
I20240507 08:11:58.922159 5840 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.926523 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 279e95d9804c44eaa2c23a4ec02c84ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.928210 5840 tablet_bootstrap.cc:654] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.928225 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.928261 6350 tablet.cc:1722] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240507 08:11:58.928910 5378 leader_election.cc:341] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.929562 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3769eb670ff442e2ad2fa5b6a934512a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.930254 5841 ts_tablet_manager.cc:1888] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.930418 5841 log.cc:1196] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:11:58.930768 5841 ts_tablet_manager.cc:1909] T c84a2018f1c442e2a4dfc1c6b9c5ceef P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.931372 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (table default.Table_b1add7221be940138ee5b9e2e796e92c [id=70d5dddc40a3453bb64b8019c2a18046]) successfully deleted
I20240507 08:11:58.933687 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:58.934928 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 53fb38b567c5455ebc441e5f71db694f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53fb38b567c5455ebc441e5f71db694f already in progress: creating tablet
I20240507 08:11:58.940048 5839 tablet_bootstrap.cc:492] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:58.941618 6377 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:58.942342 6378 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:58.942778 5841 tablet_replica.cc:330] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.943466 5841 raft_consensus.cc:2229] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.944973 5843 tablet_bootstrap.cc:654] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:58.947451 5839 ts_tablet_manager.cc:1367] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.613s user 0.003s sys 0.026s
I20240507 08:11:58.953625 5841 raft_consensus.cc:2258] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.956344 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:11:58.957533 5841 ts_tablet_manager.cc:1875] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.961831 5837 ts_tablet_manager.cc:1888] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.963124 5837 log.cc:1196] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/60bc7bc06e494adcb6a8641b50b90e6c
I20240507 08:11:58.965627 6376 raft_consensus.cc:482] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.965870 6376 raft_consensus.cc:504] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.966274 5837 ts_tablet_manager.cc:1909] T 60bc7bc06e494adcb6a8641b50b90e6c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:58.966976 6376 leader_election.cc:288] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:58.967308 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ce11e4e46b345dba1e5e1a6f1af3dd4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:58.968271 5656 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 6f6670e99e4a4442801d42950eb7ab79
I20240507 08:11:58.972287 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59dbc2eaa45d4da58951b45423f284fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.972709 5841 ts_tablet_manager.cc:1888] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.973402 5841 log.cc:1196] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/88a2cfb5f046434c8340234a447a7e65
I20240507 08:11:58.973858 5841 ts_tablet_manager.cc:1909] T 88a2cfb5f046434c8340234a447a7e65 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:58.974452 5840 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:58.974673 5840 ts_tablet_manager.cc:1367] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.003s
I20240507 08:11:58.975576 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f6670e99e4a4442801d42950eb7ab79" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:11:58.975704 5840 raft_consensus.cc:348] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.976125 5840 raft_consensus.cc:374] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:11:58.976150 5654 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:58.976343 5654 leader_election.cc:302] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:58.976841 6376 raft_consensus.cc:2726] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:58.977104 5840 raft_consensus.cc:729] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.977474 5840 consensus_queue.cc:260] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.979527 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 88a2cfb5f046434c8340234a447a7e65 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
I20240507 08:11:58.979702 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fb38b567c5455ebc441e5f71db694f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:58.979862 5841 tablet_replica.cc:330] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:58.980082 5841 raft_consensus.cc:2229] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.980253 5841 raft_consensus.cc:2258] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.981282 5840 ts_tablet_manager.cc:1398] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:11:58.981642 5840 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:58.982131 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 60bc7bc06e494adcb6a8641b50b90e6c (table default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]) successfully deleted
I20240507 08:11:58.983355 5839 raft_consensus.cc:348] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.983590 5839 raft_consensus.cc:374] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:58.983702 5995 raft_consensus.cc:482] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:58.983872 5995 raft_consensus.cc:504] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.984232 5841 ts_tablet_manager.cc:1875] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.984817 5995 leader_election.cc:288] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.985694 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364903504aa14d08b050a0cf83b82c71" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:58.986115 5718 raft_consensus.cc:2454] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:11:58.986554 5377 leader_election.cc:302] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:58.986944 5995 raft_consensus.cc:2781] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:11:58.987073 5995 raft_consensus.cc:482] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:11:58.987185 5995 raft_consensus.cc:3037] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:58.988054 5837 tablet_replica.cc:330] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:58.988282 5837 raft_consensus.cc:2229] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:58.988436 5837 raft_consensus.cc:2258] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:58.991503 5995 raft_consensus.cc:504] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:58.992141 5837 ts_tablet_manager.cc:1875] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:58.992612 5839 raft_consensus.cc:729] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:58.992902 5839 consensus_queue.cc:260] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:58.994342 5995 leader_election.cc:288] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:58.994601 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364903504aa14d08b050a0cf83b82c71" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:11:58.994781 5718 raft_consensus.cc:3037] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:11:58.995417 5841 ts_tablet_manager.cc:1888] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:58.995571 5841 log.cc:1196] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d9ff5e1b6a9a476c8295a0d0ee85aecc
I20240507 08:11:58.998379 5718 raft_consensus.cc:2454] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:11:58.998845 5377 leader_election.cc:302] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:11:59.000995 5840 tablet_bootstrap.cc:654] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.002082 5995 raft_consensus.cc:2781] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:11:59.002655 5995 raft_consensus.cc:686] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:11:59.002988 5995 consensus_queue.cc:237] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.004531 5841 ts_tablet_manager.cc:1909] T d9ff5e1b6a9a476c8295a0d0ee85aecc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:59.007655 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (table default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]) successfully deleted
I20240507 08:11:59.009645 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364903504aa14d08b050a0cf83b82c71" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:59.014281 5378 leader_election.cc:341] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:59.018332 5837 ts_tablet_manager.cc:1888] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.025645 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_e85ccc16434742ac812466c315098b9a [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]). Expected version 3 got 2
I20240507 08:11:59.025885 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:11:59.026643 5260 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_e85ccc16434742ac812466c315098b9a [id=42e0c3835e194cd6a54f6ad7d8e4f5ed] alter complete (version 3)
W20240507 08:11:59.028257 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 167ee9cbc9174b82824c468e098472a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167ee9cbc9174b82824c468e098472a6 already in progress: creating tablet
W20240507 08:11:59.033530 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4e5439844f384513b4719cfe4f4e5d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5439844f384513b4719cfe4f4e5d9b already in progress: creating tablet
I20240507 08:11:59.035256 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_33512c5a59404199a0ac2d5ff4e78bba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.035708 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3e1a43d3bc4a4f6c8f082e4f0525625f" } new_table_name: "default.Table_71341b71104946439f1c74a9dc7b9c3d" modify_external_catalogs: true
W20240507 08:11:59.035933 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33512c5a59404199a0ac2d5ff4e78bba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.036996 5841 tablet_replica.cc:330] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:59.037289 5841 raft_consensus.cc:2229] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.037523 5841 raft_consensus.cc:2258] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.037446 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_81faed45055c4c6c9d1cb2efbd2993a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.037933 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81faed45055c4c6c9d1cb2efbd2993a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.038216 5841 ts_tablet_manager.cc:1875] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.038753 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2f9027dff4a546ffb9d51a689687e01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.039211 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f9027dff4a546ffb9d51a689687e01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.040289 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.041442 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 (DEFAULT_TABLE table=default.Table_287233b419e646b58b404eb7190a3ac6 [id=28572caf37234d87953c7b84a1bb70d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.041872 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310c601abead42e0bd2a2df1a0e16ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.042973 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_105a7078d5c648b7ac16943b060dbcca" } modify_external_catalogs: true
I20240507 08:11:59.043161 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_105a7078d5c648b7ac16943b060dbcca" } modify_external_catalogs: true
I20240507 08:11:59.044580 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d149e0b2a5ab44a2ba2f81157a507967" } modify_external_catalogs: true
I20240507 08:11:59.044747 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d149e0b2a5ab44a2ba2f81157a507967" } modify_external_catalogs: true
I20240507 08:11:59.046135 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9685ac4fffc44198ac025a3ca19e81dc" } modify_external_catalogs: true
I20240507 08:11:59.046324 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9685ac4fffc44198ac025a3ca19e81dc" } modify_external_catalogs: true
I20240507 08:11:59.053122 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b (DEFAULT_TABLE table=default.Table_5b058aadbe304a9b9c8ffd023c1a0470 [id=63a4899fe56843909eaca9613c842b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.054183 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6450d3d2e7924ad2b72e8bd91378282b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.059625 6383 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c202eef7f3466fa3e703644b40879d)
I20240507 08:11:59.060014 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c202eef7f3466fa3e703644b40879d. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:59.060014 6383 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:11:59.060331 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:11:59.060772 6268 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:34350 (request call id 0) took 1345 ms. Trace:
I20240507 08:11:59.061460 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35f061a2f58c4b3fa505b7b2adf6e66d" } new_table_name: "default.Table_3ad8e609d07646e49e01cc55ba8ee969" modify_external_catalogs: true
I20240507 08:11:59.064286 5841 ts_tablet_manager.cc:1888] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.064414 5841 log.cc:1196] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5e53409fe13e4a9585dbba648a25cc07
I20240507 08:11:59.064862 5841 ts_tablet_manager.cc:1909] T 5e53409fe13e4a9585dbba648a25cc07 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:59.067796 5841 tablet_replica.cc:330] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:59.067994 5841 raft_consensus.cc:2229] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.068133 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5e53409fe13e4a9585dbba648a25cc07 (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:11:59.068199 5841 raft_consensus.cc:2258] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.070878 6383 leader_election.cc:288] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a91047a8a5414eff9c06ca9ca2b548e8 (127.4.246.254:37179), 94c202eef7f3466fa3e703644b40879d (127.4.246.252:45297)
I20240507 08:11:59.072172 5841 ts_tablet_manager.cc:1875] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.073962 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:11:59.075781 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 5643e8bc0add4df885536ff947d357c6 (DEFAULT_TABLE table=default.Table_483c951eac554e4c9a5a5f6e16e2b109 [id=fe2d32478c5841c9a247da0c0f742ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.076705 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5643e8bc0add4df885536ff947d357c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.082866 5420 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a (DEFAULT_TABLE table=default.Table_0357417146364abdb06bb955b43a5fe2 [id=643f03e077d241338ba577901b59c01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.083411 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.090029 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 3b16a81588394d28945331e233e1f1db (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.090596 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b16a81588394d28945331e233e1f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.091028 5841 ts_tablet_manager.cc:1888] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.091221 5841 log.cc:1196] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8e3ffaf1781e4b108f305346476905be
I20240507 08:11:59.091704 5841 ts_tablet_manager.cc:1909] T 8e3ffaf1781e4b108f305346476905be P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:59.098917 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc49fd6841c420288aa1df3950a088b (DEFAULT_TABLE table=default.Table_8c0ea847a8d142b3a28ac68403f103ab [id=0beae72e42824f1e969a306a8d5658d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.099462 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc49fd6841c420288aa1df3950a088b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.028630 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 5643e8bc0add4df885536ff947d357c6 (DEFAULT_TABLE table=default.Table_483c951eac554e4c9a5a5f6e16e2b109 [id=fe2d32478c5841c9a247da0c0f742ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.029000 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc49fd6841c420288aa1df3950a088b (DEFAULT_TABLE table=default.Table_8c0ea847a8d142b3a28ac68403f103ab [id=0beae72e42824f1e969a306a8d5658d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.029320 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b (DEFAULT_TABLE table=default.Table_5b058aadbe304a9b9c8ffd023c1a0470 [id=63a4899fe56843909eaca9613c842b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.029810 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 (DEFAULT_TABLE table=default.Table_287233b419e646b58b404eb7190a3ac6 [id=28572caf37234d87953c7b84a1bb70d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.039489 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 3b16a81588394d28945331e233e1f1db (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.082188 5683 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a (DEFAULT_TABLE table=default.Table_0357417146364abdb06bb955b43a5fe2 [id=643f03e077d241338ba577901b59c01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.105795 5840 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:59.092375 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8e3ffaf1781e4b108f305346476905be (table default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]) successfully deleted
I20240507 08:11:59.113867 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5643e8bc0add4df885536ff947d357c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.114295 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 072702ef2a6d43248132da6aa3a2d493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072702ef2a6d43248132da6aa3a2d493 already in progress: creating tablet
W20240507 08:11:59.114554 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0d617213b959452ab6519809e56db128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d617213b959452ab6519809e56db128 already in progress: creating tablet
I20240507 08:11:59.114737 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6450d3d2e7924ad2b72e8bd91378282b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.114761 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 831153d2b89e43f8b08b086f7a8f3a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 831153d2b89e43f8b08b086f7a8f3a54 already in progress: creating tablet
W20240507 08:11:59.115006 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef because the tablet was not found. No further retry: Not found: Tablet not found: c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:11:59.115289 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 379807692b6147599ed71578869a52e2 (DEFAULT_TABLE table=default.Table_269bbea2c2e04830a74ce31662db3c24 [id=93ae351544674a33a7b09d2d8f2ee98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.111279 5692 tablet_service.cc:1467] Processing CreateTablet for tablet c4b027513fe34b5686f396cff8ff6c25 (DEFAULT_TABLE table=default.Table_b2ab50d2ef764827aa64c3a4eaf14190 [id=7517e4b8f3094233aaab86f82f18311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.116245 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379807692b6147599ed71578869a52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.115814 5416 tablet_service.cc:1467] Processing CreateTablet for tablet c4b027513fe34b5686f396cff8ff6c25 (DEFAULT_TABLE table=default.Table_b2ab50d2ef764827aa64c3a4eaf14190 [id=7517e4b8f3094233aaab86f82f18311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.117844 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b027513fe34b5686f396cff8ff6c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.120460 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "364903504aa14d08b050a0cf83b82c71" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:11:59.124110 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef77ef4f474107bf31395f5f30c12f (DEFAULT_TABLE table=default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9 [id=dda77caf71b84b32aa7166d371e33f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.125079 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef77ef4f474107bf31395f5f30c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.127187 5417 tablet_service.cc:1467] Processing CreateTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 (DEFAULT_TABLE table=default.Table_e1b850d9c6864653b0ca7439322d02a7 [id=148ae60af9ab4151b1c1f169da0f415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.127317 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 80c892dab4414411acce4dd160079672 (DEFAULT_TABLE table=default.Table_15e48d810d164ffba2efc7a11fc51244 [id=59986073ff1148b8bf0693fb7749cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.127761 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8935f7c20ed43a69e9cfcb28fe5d023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.129034 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c892dab4414411acce4dd160079672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.130277 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 2717b2d928674c17809ef4b0141e352e (DEFAULT_TABLE table=default.Table_225c3dd465f0444a9688b5c17d613ad5 [id=a8465649fca545869d680bdd519bbdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.130753 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2717b2d928674c17809ef4b0141e352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.131567 5421 tablet_service.cc:1467] Processing CreateTablet for tablet c053869c840e4773a3c8e07a8b426b28 (DEFAULT_TABLE table=default.Table_f0ed0e76d4c746c189b43ebcf1c271b5 [id=f10084b546a741d6887431dfd9167c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.131947 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c053869c840e4773a3c8e07a8b426b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.134184 5410 tablet_service.cc:1467] Processing CreateTablet for tablet e561c811a1de48f9bc91175a3d22805a (DEFAULT_TABLE table=default.Table_4cf5e458eb7e4fe4b1a953b18d982297 [id=e8216e827ce74d71ad7bce5f75f7fe31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.134557 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e561c811a1de48f9bc91175a3d22805a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.140251 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 379807692b6147599ed71578869a52e2 (DEFAULT_TABLE table=default.Table_269bbea2c2e04830a74ce31662db3c24 [id=93ae351544674a33a7b09d2d8f2ee98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.141297 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc49fd6841c420288aa1df3950a088b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.141367 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef77ef4f474107bf31395f5f30c12f (DEFAULT_TABLE table=default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9 [id=dda77caf71b84b32aa7166d371e33f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.142689 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef77ef4f474107bf31395f5f30c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.145933 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.151273 5690 tablet_service.cc:1467] Processing CreateTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 (DEFAULT_TABLE table=default.Table_e1b850d9c6864653b0ca7439322d02a7 [id=148ae60af9ab4151b1c1f169da0f415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.152264 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 80c892dab4414411acce4dd160079672 (DEFAULT_TABLE table=default.Table_15e48d810d164ffba2efc7a11fc51244 [id=59986073ff1148b8bf0693fb7749cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.154181 5681 tablet_service.cc:1467] Processing CreateTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f (DEFAULT_TABLE table=default.Table_6828726f0a3b436faf6ac20342edff29 [id=97ff496559474b24b15ab9935c4860dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.154924 5687 tablet_service.cc:1467] Processing CreateTablet for tablet c053869c840e4773a3c8e07a8b426b28 (DEFAULT_TABLE table=default.Table_f0ed0e76d4c746c189b43ebcf1c271b5 [id=f10084b546a741d6887431dfd9167c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.156349 6350 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:11:59.168682 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 3dc49fd6841c420288aa1df3950a088b (DEFAULT_TABLE table=default.Table_8c0ea847a8d142b3a28ac68403f103ab [id=0beae72e42824f1e969a306a8d5658d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.171730 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b (DEFAULT_TABLE table=default.Table_5b058aadbe304a9b9c8ffd023c1a0470 [id=63a4899fe56843909eaca9613c842b21]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:59.172629 5378 leader_election.cc:341] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:59.174484 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 3b16a81588394d28945331e233e1f1db (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.109303 6396 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20240507 08:11:59.177316 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3dc49fd6841c420288aa1df3950a088b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.181820 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b16a81588394d28945331e233e1f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.182652 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6450d3d2e7924ad2b72e8bd91378282b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.184026 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379807692b6147599ed71578869a52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.184273 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c053869c840e4773a3c8e07a8b426b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.184399 5689 tablet_service.cc:1467] Processing CreateTablet for tablet e561c811a1de48f9bc91175a3d22805a (DEFAULT_TABLE table=default.Table_4cf5e458eb7e4fe4b1a953b18d982297 [id=e8216e827ce74d71ad7bce5f75f7fe31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.184592 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b027513fe34b5686f396cff8ff6c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.184794 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 2717b2d928674c17809ef4b0141e352e (DEFAULT_TABLE table=default.Table_225c3dd465f0444a9688b5c17d613ad5 [id=a8465649fca545869d680bdd519bbdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.184814 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c892dab4414411acce4dd160079672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.185344 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8935f7c20ed43a69e9cfcb28fe5d023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.185621 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b16a81588394d28945331e233e1f1db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.185878 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c8db73b8cb4212b59d40bd6864e23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.186074 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310c601abead42e0bd2a2df1a0e16ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.194609 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2717b2d928674c17809ef4b0141e352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.199327 5839 ts_tablet_manager.cc:1398] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.252s user 0.022s sys 0.026s
I20240507 08:11:59.199884 5839 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:59.176501 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 5643e8bc0add4df885536ff947d357c6 (DEFAULT_TABLE table=default.Table_483c951eac554e4c9a5a5f6e16e2b109 [id=fe2d32478c5841c9a247da0c0f742ef5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.203088 5837 log.cc:1196] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/6f6670e99e4a4442801d42950eb7ab79
I20240507 08:11:59.209544 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5643e8bc0add4df885536ff947d357c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.211911 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 (DEFAULT_TABLE table=default.Table_287233b419e646b58b404eb7190a3ac6 [id=28572caf37234d87953c7b84a1bb70d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.212436 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 310c601abead42e0bd2a2df1a0e16ec2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.215376 5559 tablet_service.cc:1467] Processing CreateTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a (DEFAULT_TABLE table=default.Table_0357417146364abdb06bb955b43a5fe2 [id=643f03e077d241338ba577901b59c01e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.215723 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.215811 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e561c811a1de48f9bc91175a3d22805a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.217551 5840 ts_tablet_manager.cc:1367] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.132s user 0.006s sys 0.001s
I20240507 08:11:59.220300 5561 tablet_service.cc:1467] Processing CreateTablet for tablet c4b027513fe34b5686f396cff8ff6c25 (DEFAULT_TABLE table=default.Table_b2ab50d2ef764827aa64c3a4eaf14190 [id=7517e4b8f3094233aaab86f82f18311a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.221930 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b027513fe34b5686f396cff8ff6c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.225405 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 379807692b6147599ed71578869a52e2 (DEFAULT_TABLE table=default.Table_269bbea2c2e04830a74ce31662db3c24 [id=93ae351544674a33a7b09d2d8f2ee98d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.225806 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 379807692b6147599ed71578869a52e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.232020 5131 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 601 } } ignore_live_leader: false dest_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" is_pre_election: true
I20240507 08:11:59.236553 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 80c892dab4414411acce4dd160079672 (DEFAULT_TABLE table=default.Table_15e48d810d164ffba2efc7a11fc51244 [id=59986073ff1148b8bf0693fb7749cdc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.244304 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 96ef77ef4f474107bf31395f5f30c12f (DEFAULT_TABLE table=default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9 [id=dda77caf71b84b32aa7166d371e33f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.060911 6268 rpcz_store.cc:276] 0507 08:11:57.715025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.715086 (+ 61us) service_pool.cc:225] Handling call
0507 08:11:58.301975 (+586889us) raft_consensus.cc:1457] Updating replica for 581 ops
0507 08:11:58.411892 (+109917us) raft_consensus.cc:1520] Early marking committed up to index 0
0507 08:11:58.411905 (+ 13us) raft_consensus.cc:1525] Triggering prepare for 581 ops
0507 08:11:58.412608 (+ 703us) write_op.cc:270] Start()
0507 08:11:58.412690 (+ 82us) write_op.cc:276] Timestamp: P: 1715069509392731 usec, L: 0
0507 08:11:58.423980 (+ 11290us) write_op.cc:270] Start()
0507 08:11:58.424024 (+ 44us) write_op.cc:276] Timestamp: P: 1715069509404453 usec, L: 0
0507 08:11:58.424285 (+ 261us) write_op.cc:270] Start()
0507 08:11:58.424319 (+ 34us) write_op.cc:276] Timestamp: P: 1715069509405162 usec, L: 0
0507 08:11:58.424477 (+ 158us) write_op.cc:270] Start()
0507 08:11:58.424498 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509405853 usec, L: 0
0507 08:11:58.424636 (+ 138us) write_op.cc:270] Start()
0507 08:11:58.424656 (+ 20us) write_op.cc:276] Timestamp: P: 1715069509406549 usec, L: 0
0507 08:11:58.424782 (+ 126us) write_op.cc:270] Start()
0507 08:11:58.424800 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509450524 usec, L: 0
0507 08:11:58.424918 (+ 118us) write_op.cc:270] Start()
0507 08:11:58.424990 (+ 72us) write_op.cc:276] Timestamp: P: 1715069509532167 usec, L: 0
0507 08:11:58.425159 (+ 169us) write_op.cc:270] Start()
0507 08:11:58.425182 (+ 23us) write_op.cc:276] Timestamp: P: 1715069509533348 usec, L: 0
0507 08:11:58.425326 (+ 144us) write_op.cc:270] Start()
0507 08:11:58.425349 (+ 23us) write_op.cc:276] Timestamp: P: 1715069509534982 usec, L: 0
0507 08:11:58.425463 (+ 114us) write_op.cc:270] Start()
0507 08:11:58.425506 (+ 43us) write_op.cc:276] Timestamp: P: 1715069509535566 usec, L: 0
0507 08:11:58.425728 (+ 222us) write_op.cc:270] Start()
0507 08:11:58.425748 (+ 20us) write_op.cc:276] Timestamp: P: 1715069509552211 usec, L: 0
0507 08:11:58.612107 (+186359us) write_op.cc:270] Start()
0507 08:11:58.620679 (+ 8572us) write_op.cc:276] Timestamp: P: 1715069509552743 usec, L: 0
0507 08:11:58.626900 (+ 6221us) write_op.cc:270] Start()
0507 08:11:58.626935 (+ 35us) write_op.cc:276] Timestamp: P: 1715069509553278 usec, L: 0
0507 08:11:58.627113 (+ 178us) write_op.cc:270] Start()
0507 08:11:58.627136 (+ 23us) write_op.cc:276] Timestamp: P: 1715069509566533 usec, L: 0
0507 08:11:58.627332 (+ 196us) write_op.cc:270] Start()
0507 08:11:58.627367 (+ 35us) write_op.cc:276] Timestamp: P: 1715069509567124 usec, L: 0
0507 08:11:58.627568 (+ 201us) write_op.cc:270] Start()
0507 08:11:58.627593 (+ 25us) write_op.cc:276] Timestamp: P: 1715069509567636 usec, L: 0
0507 08:11:58.627745 (+ 152us) write_op.cc:270] Start()
0507 08:11:58.627779 (+ 34us) write_op.cc:276] Timestamp: P: 1715069509570817 usec, L: 0
0507 08:11:58.627967 (+ 188us) write_op.cc:270] Start()
0507 08:11:58.627989 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509590632 usec, L: 0
0507 08:11:58.628143 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.628164 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509591222 usec, L: 0
0507 08:11:58.628299 (+ 135us) write_op.cc:270] Start()
0507 08:11:58.628318 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509591842 usec, L: 0
0507 08:11:58.628483 (+ 165us) write_op.cc:270] Start()
0507 08:11:58.628504 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509605893 usec, L: 0
0507 08:11:58.628679 (+ 175us) write_op.cc:270] Start()
0507 08:11:58.628700 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509606722 usec, L: 0
0507 08:11:58.628838 (+ 138us) write_op.cc:270] Start()
0507 08:11:58.628863 (+ 25us) write_op.cc:276] Timestamp: P: 1715069509607486 usec, L: 0
0507 08:11:58.628994 (+ 131us) write_op.cc:270] Start()
0507 08:11:58.629013 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509609950 usec, L: 0
0507 08:11:58.629167 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.629186 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509610686 usec, L: 0
0507 08:11:58.629303 (+ 117us) write_op.cc:270] Start()
0507 08:11:58.629321 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509624274 usec, L: 0
0507 08:11:58.629449 (+ 128us) write_op.cc:270] Start()
0507 08:11:58.629468 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509624886 usec, L: 0
0507 08:11:58.629649 (+ 181us) write_op.cc:270] Start()
0507 08:11:58.629671 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509625445 usec, L: 0
0507 08:11:58.629850 (+ 179us) write_op.cc:270] Start()
0507 08:11:58.629871 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509649938 usec, L: 0
0507 08:11:58.630012 (+ 141us) write_op.cc:270] Start()
0507 08:11:58.630036 (+ 24us) write_op.cc:276] Timestamp: P: 1715069509656696 usec, L: 0
0507 08:11:58.630171 (+ 135us) write_op.cc:270] Start()
0507 08:11:58.630207 (+ 36us) write_op.cc:276] Timestamp: P: 1715069509664018 usec, L: 0
0507 08:11:58.630354 (+ 147us) write_op.cc:270] Start()
0507 08:11:58.630377 (+ 23us) write_op.cc:276] Timestamp: P: 1715069509665216 usec, L: 0
0507 08:11:58.630573 (+ 196us) write_op.cc:270] Start()
0507 08:11:58.630598 (+ 25us) write_op.cc:276] Timestamp: P: 1715069509667443 usec, L: 0
0507 08:11:58.630734 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.630762 (+ 28us) write_op.cc:276] Timestamp: P: 1715069509678462 usec, L: 0
0507 08:11:58.630914 (+ 152us) write_op.cc:270] Start()
0507 08:11:58.630936 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509763448 usec, L: 0
0507 08:11:58.631069 (+ 133us) write_op.cc:270] Start()
0507 08:11:58.631091 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509771000 usec, L: 0
0507 08:11:58.631240 (+ 149us) write_op.cc:270] Start()
0507 08:11:58.631276 (+ 36us) write_op.cc:276] Timestamp: P: 1715069509774254 usec, L: 0
0507 08:11:58.631443 (+ 167us) write_op.cc:270] Start()
0507 08:11:58.631462 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509787202 usec, L: 0
0507 08:11:58.631592 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.631610 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509802696 usec, L: 0
0507 08:11:58.631750 (+ 140us) write_op.cc:270] Start()
0507 08:11:58.631768 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509803269 usec, L: 0
0507 08:11:58.631923 (+ 155us) write_op.cc:270] Start()
0507 08:11:58.631947 (+ 24us) write_op.cc:276] Timestamp: P: 1715069509806316 usec, L: 0
0507 08:11:58.632094 (+ 147us) write_op.cc:270] Start()
0507 08:11:58.632116 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509811186 usec, L: 0
0507 08:11:58.632248 (+ 132us) write_op.cc:270] Start()
0507 08:11:58.632273 (+ 25us) write_op.cc:276] Timestamp: P: 1715069509823204 usec, L: 0
0507 08:11:58.632393 (+ 120us) write_op.cc:270] Start()
0507 08:11:58.632418 (+ 25us) write_op.cc:276] Timestamp: P: 1715069509825765 usec, L: 0
0507 08:11:58.632572 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.632590 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509828600 usec, L: 0
0507 08:11:58.632726 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.632747 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509850530 usec, L: 0
0507 08:11:58.632882 (+ 135us) write_op.cc:270] Start()
0507 08:11:58.632903 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509851146 usec, L: 0
0507 08:11:58.633155 (+ 252us) write_op.cc:270] Start()
0507 08:11:58.633178 (+ 23us) write_op.cc:276] Timestamp: P: 1715069509851743 usec, L: 0
0507 08:11:58.633309 (+ 131us) write_op.cc:270] Start()
0507 08:11:58.633329 (+ 20us) write_op.cc:276] Timestamp: P: 1715069509863262 usec, L: 0
0507 08:11:58.633472 (+ 143us) write_op.cc:270] Start()
0507 08:11:58.633496 (+ 24us) write_op.cc:276] Timestamp: P: 1715069509882607 usec, L: 0
0507 08:11:58.633757 (+ 261us) write_op.cc:270] Start()
0507 08:11:58.633783 (+ 26us) write_op.cc:276] Timestamp: P: 1715069509891196 usec, L: 0
0507 08:11:58.633941 (+ 158us) write_op.cc:270] Start()
0507 08:11:58.633962 (+ 21us) write_op.cc:276] Timestamp: P: 1715069509895637 usec, L: 0
0507 08:11:58.634104 (+ 142us) write_op.cc:270] Start()
0507 08:11:58.634126 (+ 22us) write_op.cc:276] Timestamp: P: 1715069509902406 usec, L: 0
0507 08:11:58.634255 (+ 129us) write_op.cc:270] Start()
0507 08:11:58.634272 (+ 17us) write_op.cc:276] Timestamp: P: 1715069509903472 usec, L: 0
0507 08:11:58.634366 (+ 94us) write_op.cc:270] Start()
0507 08:11:58.634381 (+ 15us) write_op.cc:276] Timestamp: P: 1715069509904553 usec, L: 0
0507 08:11:58.634486 (+ 105us) write_op.cc:270] Start()
0507 08:11:58.634505 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509946444 usec, L: 0
0507 08:11:58.634629 (+ 124us) write_op.cc:270] Start()
0507 08:11:58.634647 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509947588 usec, L: 0
0507 08:11:58.634749 (+ 102us) write_op.cc:270] Start()
0507 08:11:58.634768 (+ 19us) write_op.cc:276] Timestamp: P: 1715069509952044 usec, L: 0
0507 08:11:58.634887 (+ 119us) write_op.cc:270] Start()
0507 08:11:58.634905 (+ 18us) write_op.cc:276] Timestamp: P: 1715069509952637 usec, L: 0
0507 08:11:58.635018 (+ 113us) write_op.cc:270] Start()
0507 08:11:58.635057 (+ 39us) write_op.cc:276] Timestamp: P: 1715069509998748 usec, L: 0
0507 08:11:58.635177 (+ 120us) write_op.cc:270] Start()
0507 08:11:58.635193 (+ 16us) write_op.cc:276] Timestamp: P: 1715069510024873 usec, L: 0
0507 08:11:58.635310 (+ 117us) write_op.cc:270] Start()
0507 08:11:58.635334 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510035379 usec, L: 0
0507 08:11:58.635455 (+ 121us) write_op.cc:270] Start()
0507 08:11:58.635487 (+ 32us) write_op.cc:276] Timestamp: P: 1715069510035963 usec, L: 0
0507 08:11:58.635633 (+ 146us) write_op.cc:270] Start()
0507 08:11:58.635658 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510053716 usec, L: 0
0507 08:11:58.635849 (+ 191us) write_op.cc:270] Start()
0507 08:11:58.635873 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510068344 usec, L: 0
0507 08:11:58.636037 (+ 164us) write_op.cc:270] Start()
0507 08:11:58.636062 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510079259 usec, L: 0
0507 08:11:58.636200 (+ 138us) write_op.cc:270] Start()
0507 08:11:58.636223 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510081092 usec, L: 0
0507 08:11:58.636402 (+ 179us) write_op.cc:270] Start()
0507 08:11:58.636424 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510100261 usec, L: 0
0507 08:11:58.636623 (+ 199us) write_op.cc:270] Start()
0507 08:11:58.636646 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510101998 usec, L: 0
0507 08:11:58.636809 (+ 163us) write_op.cc:270] Start()
0507 08:11:58.636829 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510102510 usec, L: 0
0507 08:11:58.636964 (+ 135us) write_op.cc:270] Start()
0507 08:11:58.636985 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510103036 usec, L: 0
0507 08:11:58.637143 (+ 158us) write_op.cc:270] Start()
0507 08:11:58.637167 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510103502 usec, L: 0
0507 08:11:58.637334 (+ 167us) write_op.cc:270] Start()
0507 08:11:58.637361 (+ 27us) write_op.cc:276] Timestamp: P: 1715069510120524 usec, L: 0
0507 08:11:58.637522 (+ 161us) write_op.cc:270] Start()
0507 08:11:58.637544 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510132361 usec, L: 0
0507 08:11:58.637712 (+ 168us) write_op.cc:270] Start()
0507 08:11:58.637734 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510134252 usec, L: 0
0507 08:11:58.637889 (+ 155us) write_op.cc:270] Start()
0507 08:11:58.637914 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510181807 usec, L: 0
0507 08:11:58.638075 (+ 161us) write_op.cc:270] Start()
0507 08:11:58.638093 (+ 18us) write_op.cc:276] Timestamp: P: 1715069510182521 usec, L: 0
0507 08:11:58.638254 (+ 161us) write_op.cc:270] Start()
0507 08:11:58.638280 (+ 26us) write_op.cc:276] Timestamp: P: 1715069510204470 usec, L: 0
0507 08:11:58.638617 (+ 337us) write_op.cc:270] Start()
0507 08:11:58.638679 (+ 62us) write_op.cc:276] Timestamp: P: 1715069510205415 usec, L: 0
0507 08:11:58.638891 (+ 212us) write_op.cc:270] Start()
0507 08:11:58.638919 (+ 28us) write_op.cc:276] Timestamp: P: 1715069510217192 usec, L: 0
0507 08:11:58.639081 (+ 162us) write_op.cc:270] Start()
0507 08:11:58.639099 (+ 18us) write_op.cc:276] Timestamp: P: 1715069510239850 usec, L: 0
0507 08:11:58.639253 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.639273 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510247219 usec, L: 0
0507 08:11:58.639435 (+ 162us) write_op.cc:270] Start()
0507 08:11:58.639455 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510275388 usec, L: 0
0507 08:11:58.639713 (+ 258us) write_op.cc:270] Start()
0507 08:11:58.639744 (+ 31us) write_op.cc:276] Timestamp: P: 1715069510277999 usec, L: 0
0507 08:11:58.639976 (+ 232us) write_op.cc:270] Start()
0507 08:11:58.640001 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510289421 usec, L: 0
0507 08:11:58.640141 (+ 140us) write_op.cc:270] Start()
0507 08:11:58.640164 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510297300 usec, L: 0
0507 08:11:58.640312 (+ 148us) write_op.cc:270] Start()
0507 08:11:58.640335 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510328615 usec, L: 0
0507 08:11:58.640478 (+ 143us) write_op.cc:270] Start()
0507 08:11:58.640499 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510344402 usec, L: 0
0507 08:11:58.640658 (+ 159us) write_op.cc:270] Start()
0507 08:11:58.640691 (+ 33us) write_op.cc:276] Timestamp: P: 1715069510353176 usec, L: 0
0507 08:11:58.640847 (+ 156us) write_op.cc:270] Start()
0507 08:11:58.640881 (+ 34us) write_op.cc:276] Timestamp: P: 1715069510363746 usec, L: 0
0507 08:11:58.641092 (+ 211us) write_op.cc:270] Start()
0507 08:11:58.641123 (+ 31us) write_op.cc:276] Timestamp: P: 1715069510366276 usec, L: 0
0507 08:11:58.641319 (+ 196us) write_op.cc:270] Start()
0507 08:11:58.641345 (+ 26us) write_op.cc:276] Timestamp: P: 1715069510406564 usec, L: 0
0507 08:11:58.641510 (+ 165us) write_op.cc:270] Start()
0507 08:11:58.641532 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510408393 usec, L: 0
0507 08:11:58.641693 (+ 161us) write_op.cc:270] Start()
0507 08:11:58.641717 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510410345 usec, L: 0
0507 08:11:58.641866 (+ 149us) write_op.cc:270] Start()
0507 08:11:58.641900 (+ 34us) write_op.cc:276] Timestamp: P: 1715069510422098 usec, L: 0
0507 08:11:58.643010 (+ 1110us) write_op.cc:270] Start()
0507 08:11:58.643046 (+ 36us) write_op.cc:276] Timestamp: P: 1715069510423092 usec, L: 0
0507 08:11:58.643211 (+ 165us) write_op.cc:270] Start()
0507 08:11:58.643234 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510423932 usec, L: 0
0507 08:11:58.643371 (+ 137us) write_op.cc:270] Start()
0507 08:11:58.643396 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510427586 usec, L: 0
0507 08:11:58.643567 (+ 171us) write_op.cc:270] Start()
0507 08:11:58.643589 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510430261 usec, L: 0
0507 08:11:58.643720 (+ 131us) write_op.cc:270] Start()
0507 08:11:58.643744 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510432405 usec, L: 0
0507 08:11:58.643920 (+ 176us) write_op.cc:270] Start()
0507 08:11:58.643942 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510445495 usec, L: 0
0507 08:11:58.644066 (+ 124us) write_op.cc:270] Start()
0507 08:11:58.644085 (+ 19us) write_op.cc:276] Timestamp: P: 1715069510456094 usec, L: 0
0507 08:11:58.644197 (+ 112us) write_op.cc:270] Start()
0507 08:11:58.644217 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510456810 usec, L: 0
0507 08:11:58.644333 (+ 116us) write_op.cc:270] Start()
0507 08:11:58.644352 (+ 19us) write_op.cc:276] Timestamp: P: 1715069510466972 usec, L: 0
0507 08:11:58.644480 (+ 128us) write_op.cc:270] Start()
0507 08:11:58.644501 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510500042 usec, L: 0
0507 08:11:58.644657 (+ 156us) write_op.cc:270] Start()
0507 08:11:58.644682 (+ 25us) write_op.cc:276] Timestamp: P: 1715069510501787 usec, L: 0
0507 08:11:58.644841 (+ 159us) write_op.cc:270] Start()
0507 08:11:58.644861 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510504636 usec, L: 0
0507 08:11:58.644997 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.645018 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510519523 usec, L: 0
0507 08:11:58.645165 (+ 147us) write_op.cc:270] Start()
0507 08:11:58.645189 (+ 24us) write_op.cc:276] Timestamp: P: 1715069510521280 usec, L: 0
0507 08:11:58.645330 (+ 141us) write_op.cc:270] Start()
0507 08:11:58.645351 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510544193 usec, L: 0
0507 08:11:58.645598 (+ 247us) write_op.cc:270] Start()
0507 08:11:58.645628 (+ 30us) write_op.cc:276] Timestamp: P: 1715069510545347 usec, L: 0
0507 08:11:58.645806 (+ 178us) write_op.cc:270] Start()
0507 08:11:58.645832 (+ 26us) write_op.cc:276] Timestamp: P: 1715069510562930 usec, L: 0
0507 08:11:58.645977 (+ 145us) write_op.cc:270] Start()
0507 08:11:58.645999 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510582120 usec, L: 0
0507 08:11:58.646133 (+ 134us) write_op.cc:270] Start()
0507 08:11:58.646150 (+ 17us) write_op.cc:276] Timestamp: P: 1715069510596043 usec, L: 0
0507 08:11:58.646286 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.646306 (+ 20us) write_op.cc:276] Timestamp: P: 1715069510601902 usec, L: 0
0507 08:11:58.646442 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.646464 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510636898 usec, L: 0
0507 08:11:58.646825 (+ 361us) write_op.cc:270] Start()
0507 08:11:58.646856 (+ 31us) write_op.cc:276] Timestamp: P: 1715069510682601 usec, L: 0
0507 08:11:58.647027 (+ 171us) write_op.cc:270] Start()
0507 08:11:58.647053 (+ 26us) write_op.cc:276] Timestamp: P: 1715069510698761 usec, L: 0
0507 08:11:58.647198 (+ 145us) write_op.cc:270] Start()
0507 08:11:58.647215 (+ 17us) write_op.cc:276] Timestamp: P: 1715069510717469 usec, L: 0
0507 08:11:58.647359 (+ 144us) write_op.cc:270] Start()
0507 08:11:58.647382 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510726502 usec, L: 0
0507 08:11:58.647542 (+ 160us) write_op.cc:270] Start()
0507 08:11:58.647563 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510739301 usec, L: 0
0507 08:11:58.647702 (+ 139us) write_op.cc:270] Start()
0507 08:11:58.647723 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510750504 usec, L: 0
0507 08:11:58.647872 (+ 149us) write_op.cc:270] Start()
0507 08:11:58.647894 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510758990 usec, L: 0
0507 08:11:58.648068 (+ 174us) write_op.cc:270] Start()
0507 08:11:58.648095 (+ 27us) write_op.cc:276] Timestamp: P: 1715069510770555 usec, L: 0
0507 08:11:58.648235 (+ 140us) write_op.cc:270] Start()
0507 08:11:58.648253 (+ 18us) write_op.cc:276] Timestamp: P: 1715069510775380 usec, L: 0
0507 08:11:58.648454 (+ 201us) write_op.cc:270] Start()
0507 08:11:58.648477 (+ 23us) write_op.cc:276] Timestamp: P: 1715069510799198 usec, L: 0
0507 08:11:58.648632 (+ 155us) write_op.cc:270] Start()
0507 08:11:58.648658 (+ 26us) write_op.cc:276] Timestamp: P: 1715069510809791 usec, L: 0
0507 08:11:58.648812 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.648830 (+ 18us) write_op.cc:276] Timestamp: P: 1715069510812702 usec, L: 0
0507 08:11:58.649016 (+ 186us) write_op.cc:270] Start()
0507 08:11:58.649033 (+ 17us) write_op.cc:276] Timestamp: P: 1715069510838088 usec, L: 0
0507 08:11:58.649169 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.649191 (+ 22us) write_op.cc:276] Timestamp: P: 1715069510846692 usec, L: 0
0507 08:11:58.649345 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.649396 (+ 51us) write_op.cc:276] Timestamp: P: 1715069510870118 usec, L: 0
0507 08:11:58.649626 (+ 230us) write_op.cc:270] Start()
0507 08:11:58.649668 (+ 42us) write_op.cc:276] Timestamp: P: 1715069510873376 usec, L: 0
0507 08:11:58.649831 (+ 163us) write_op.cc:270] Start()
0507 08:11:58.649852 (+ 21us) write_op.cc:276] Timestamp: P: 1715069510946497 usec, L: 0
0507 08:11:58.649989 (+ 137us) write_op.cc:270] Start()
0507 08:11:58.650007 (+ 18us) write_op.cc:276] Timestamp: P: 1715069510983375 usec, L: 0
0507 08:11:58.650150 (+ 143us) write_op.cc:270] Start()
0507 08:11:58.650170 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511017312 usec, L: 0
0507 08:11:58.650302 (+ 132us) write_op.cc:270] Start()
0507 08:11:58.650323 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511030762 usec, L: 0
0507 08:11:58.650449 (+ 126us) write_op.cc:270] Start()
0507 08:11:58.650470 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511076740 usec, L: 0
0507 08:11:58.650623 (+ 153us) write_op.cc:270] Start()
0507 08:11:58.650645 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511080781 usec, L: 0
0507 08:11:58.650790 (+ 145us) write_op.cc:270] Start()
0507 08:11:58.650811 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511081934 usec, L: 0
0507 08:11:58.650946 (+ 135us) write_op.cc:270] Start()
0507 08:11:58.650968 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511083490 usec, L: 0
0507 08:11:58.651104 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.651124 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511098596 usec, L: 0
0507 08:11:58.651254 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.651272 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511099558 usec, L: 0
0507 08:11:58.651413 (+ 141us) write_op.cc:270] Start()
0507 08:11:58.651464 (+ 51us) write_op.cc:276] Timestamp: P: 1715069511100214 usec, L: 0
0507 08:11:58.651583 (+ 119us) write_op.cc:270] Start()
0507 08:11:58.651603 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511100814 usec, L: 0
0507 08:11:58.651740 (+ 137us) write_op.cc:270] Start()
0507 08:11:58.651773 (+ 33us) write_op.cc:276] Timestamp: P: 1715069511101399 usec, L: 0
0507 08:11:58.651903 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.651921 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511157135 usec, L: 0
0507 08:11:58.652032 (+ 111us) write_op.cc:270] Start()
0507 08:11:58.652052 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511175133 usec, L: 0
0507 08:11:58.652328 (+ 276us) write_op.cc:270] Start()
0507 08:11:58.652353 (+ 25us) write_op.cc:276] Timestamp: P: 1715069511176916 usec, L: 0
0507 08:11:58.652494 (+ 141us) write_op.cc:270] Start()
0507 08:11:58.652517 (+ 23us) write_op.cc:276] Timestamp: P: 1715069511178977 usec, L: 0
0507 08:11:58.652674 (+ 157us) write_op.cc:270] Start()
0507 08:11:58.652695 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511180460 usec, L: 0
0507 08:11:58.652849 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.652869 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511225861 usec, L: 0
0507 08:11:58.653014 (+ 145us) write_op.cc:270] Start()
0507 08:11:58.653036 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511236082 usec, L: 0
0507 08:11:58.653170 (+ 134us) write_op.cc:270] Start()
0507 08:11:58.653189 (+ 19us) write_op.cc:276] Timestamp: P: 1715069511247181 usec, L: 0
0507 08:11:58.653345 (+ 156us) write_op.cc:270] Start()
0507 08:11:58.653366 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511264051 usec, L: 0
0507 08:11:58.653500 (+ 134us) write_op.cc:270] Start()
0507 08:11:58.653521 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511275938 usec, L: 0
0507 08:11:58.653708 (+ 187us) write_op.cc:270] Start()
0507 08:11:58.653734 (+ 26us) write_op.cc:276] Timestamp: P: 1715069511289570 usec, L: 0
0507 08:11:58.653987 (+ 253us) write_op.cc:270] Start()
0507 08:11:58.654016 (+ 29us) write_op.cc:276] Timestamp: P: 1715069511304365 usec, L: 0
0507 08:11:58.654239 (+ 223us) write_op.cc:270] Start()
0507 08:11:58.654264 (+ 25us) write_op.cc:276] Timestamp: P: 1715069511314508 usec, L: 0
0507 08:11:58.654466 (+ 202us) write_op.cc:270] Start()
0507 08:11:58.654490 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511315156 usec, L: 0
0507 08:11:58.654611 (+ 121us) write_op.cc:270] Start()
0507 08:11:58.654637 (+ 26us) write_op.cc:276] Timestamp: P: 1715069511321730 usec, L: 0
0507 08:11:58.654793 (+ 156us) write_op.cc:270] Start()
0507 08:11:58.654819 (+ 26us) write_op.cc:276] Timestamp: P: 1715069511332382 usec, L: 0
0507 08:11:58.655061 (+ 242us) write_op.cc:270] Start()
0507 08:11:58.655091 (+ 30us) write_op.cc:276] Timestamp: P: 1715069511345509 usec, L: 0
0507 08:11:58.655245 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.655264 (+ 19us) write_op.cc:276] Timestamp: P: 1715069511355020 usec, L: 0
0507 08:11:58.655496 (+ 232us) write_op.cc:270] Start()
0507 08:11:58.655537 (+ 41us) write_op.cc:276] Timestamp: P: 1715069511355609 usec, L: 0
0507 08:11:58.655687 (+ 150us) write_op.cc:270] Start()
0507 08:11:58.655715 (+ 28us) write_op.cc:276] Timestamp: P: 1715069511362004 usec, L: 0
0507 08:11:58.655910 (+ 195us) write_op.cc:270] Start()
0507 08:11:58.655928 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511400508 usec, L: 0
0507 08:11:58.656112 (+ 184us) write_op.cc:270] Start()
0507 08:11:58.656185 (+ 73us) write_op.cc:276] Timestamp: P: 1715069511401164 usec, L: 0
0507 08:11:58.656378 (+ 193us) write_op.cc:270] Start()
0507 08:11:58.656402 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511401794 usec, L: 0
0507 08:11:58.656556 (+ 154us) write_op.cc:270] Start()
0507 08:11:58.656574 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511402320 usec, L: 0
0507 08:11:58.656711 (+ 137us) write_op.cc:270] Start()
0507 08:11:58.656736 (+ 25us) write_op.cc:276] Timestamp: P: 1715069511410512 usec, L: 0
0507 08:11:58.656908 (+ 172us) write_op.cc:270] Start()
0507 08:11:58.656932 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511411209 usec, L: 0
0507 08:11:58.657168 (+ 236us) write_op.cc:270] Start()
0507 08:11:58.657185 (+ 17us) write_op.cc:276] Timestamp: P: 1715069511420181 usec, L: 0
0507 08:11:58.657303 (+ 118us) write_op.cc:270] Start()
0507 08:11:58.657321 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511452442 usec, L: 0
0507 08:11:58.657450 (+ 129us) write_op.cc:270] Start()
0507 08:11:58.657472 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511454382 usec, L: 0
0507 08:11:58.657642 (+ 170us) write_op.cc:270] Start()
0507 08:11:58.657662 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511455293 usec, L: 0
0507 08:11:58.657800 (+ 138us) write_op.cc:270] Start()
0507 08:11:58.658116 (+ 316us) write_op.cc:276] Timestamp: P: 1715069511456157 usec, L: 0
0507 08:11:58.658287 (+ 171us) write_op.cc:270] Start()
0507 08:11:58.658309 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511456967 usec, L: 0
0507 08:11:58.658448 (+ 139us) write_op.cc:270] Start()
0507 08:11:58.658468 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511487682 usec, L: 0
0507 08:11:58.658575 (+ 107us) write_op.cc:270] Start()
0507 08:11:58.658595 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511497151 usec, L: 0
0507 08:11:58.658725 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.658744 (+ 19us) write_op.cc:276] Timestamp: P: 1715069511523399 usec, L: 0
0507 08:11:58.658872 (+ 128us) write_op.cc:270] Start()
0507 08:11:58.658890 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511540873 usec, L: 0
0507 08:11:58.659015 (+ 125us) write_op.cc:270] Start()
0507 08:11:58.659033 (+ 18us) write_op.cc:276] Timestamp: P: 1715069511557886 usec, L: 0
0507 08:11:58.664668 (+ 5635us) write_op.cc:270] Start()
0507 08:11:58.664698 (+ 30us) write_op.cc:276] Timestamp: P: 1715069511568857 usec, L: 0
0507 08:11:58.664853 (+ 155us) write_op.cc:270] Start()
0507 08:11:58.664877 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511583656 usec, L: 0
0507 08:11:58.665030 (+ 153us) write_op.cc:270] Start()
0507 08:11:58.665058 (+ 28us) write_op.cc:276] Timestamp: P: 1715069511596585 usec, L: 0
0507 08:11:58.665188 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.665208 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511611949 usec, L: 0
0507 08:11:58.665344 (+ 136us) write_op.cc:270] Start()
0507 08:11:58.665368 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511615500 usec, L: 0
0507 08:11:58.665591 (+ 223us) write_op.cc:270] Start()
0507 08:11:58.665615 (+ 24us) write_op.cc:276] Timestamp: P: 1715069511617574 usec, L: 0
0507 08:11:58.665767 (+ 152us) write_op.cc:270] Start()
0507 08:11:58.665789 (+ 22us) write_op.cc:276] Timestamp: P: 1715069511622629 usec, L: 0
0507 08:11:58.665913 (+ 124us) write_op.cc:270] Start()
0507 08:11:58.665934 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511661438 usec, L: 0
0507 08:11:58.666084 (+ 150us) write_op.cc:270] Start()
0507 08:11:58.666104 (+ 20us) write_op.cc:276] Timestamp: P: 1715069511679835 usec, L: 0
0507 08:11:58.666215 (+ 111us) write_op.cc:270] Start()
0507 08:11:58.666236 (+ 21us) write_op.cc:276] Timestamp: P: 1715069511690672 usec, L: 0
0507 08:11:58.666358 (+ 122us) write_op.cc:270] Start()
0507 08:11:58.666381 (+ 23us) write_op.cc:276] Timestamp: P: 1715069511693184 usec, L: 0
0507 08:11:58.666564 (+ 183us) write_op.cc:270] Start()
0507 08:11:58.666587 (+ 23us) write_op.cc:276] Timestamp: P: 1715069511709533 usec, L: 0
0507 08:11:58.666717 (+ 130us) write_op.cc:270] Start()
0507 08:11:58.666736 (+ 19us) write_op.cc:276] Timestamp: P: 1715069511717413 usec, L: 0
0507 08:11:58.666880 (+ 144us) write_op.cc:270] Start()
0507 08:11:58.666909 (+ 29us) write_op.cc:276] Timestamp: P: 1715069511718103 usec, L: 0
0507 08:11:58.669440 (+ 2531us) write_op.cc:270] Start()
0507 08:11:58.669472 (+ 32us) write_op.cc:276] Timestamp: P: 17150695
I20240507 08:11:59.244853 5560 tablet_service.cc:1467] Processing CreateTablet for tablet c053869c840e4773a3c8e07a8b426b28 (DEFAULT_TABLE table=default.Table_f0ed0e76d4c746c189b43ebcf1c271b5 [id=f10084b546a741d6887431dfd9167c02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.247202 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80c892dab4414411acce4dd160079672. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.251643 5555 tablet_service.cc:1467] Processing CreateTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f (DEFAULT_TABLE table=default.Table_6828726f0a3b436faf6ac20342edff29 [id=97ff496559474b24b15ab9935c4860dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.252993 5840 raft_consensus.cc:348] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.253361 5840 raft_consensus.cc:374] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.253492 5840 raft_consensus.cc:729] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.269088 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c053869c840e4773a3c8e07a8b426b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.271556 5563 tablet_service.cc:1467] Processing CreateTablet for tablet e561c811a1de48f9bc91175a3d22805a (DEFAULT_TABLE table=default.Table_4cf5e458eb7e4fe4b1a953b18d982297 [id=e8216e827ce74d71ad7bce5f75f7fe31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.271526 5553 tablet_service.cc:1467] Processing CreateTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 (DEFAULT_TABLE table=default.Table_e1b850d9c6864653b0ca7439322d02a7 [id=148ae60af9ab4151b1c1f169da0f415d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.273679 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c8db73b8cb4212b59d40bd6864e23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.273407 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 2717b2d928674c17809ef4b0141e352e (DEFAULT_TABLE table=default.Table_225c3dd465f0444a9688b5c17d613ad5 [id=a8465649fca545869d680bdd519bbdde]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.276453 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e561c811a1de48f9bc91175a3d22805a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.278771 5840 consensus_queue.cc:260] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.284909 6237 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "1cbb887175f345ada2433adaec906fd4" instance_seqno: 1715069509205254) as {username='slave'} at 127.4.246.195:33507; Asking this server to re-register.
I20240507 08:11:59.287654 5767 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:59.290138 6237 ts_manager.cc:194] Registered new tserver with Master: 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:59.296135 5839 tablet_bootstrap.cc:654] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.296301 5840 ts_tablet_manager.cc:1398] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.062s user 0.001s sys 0.001s
I20240507 08:11:59.298420 6388 raft_consensus.cc:482] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:59.298586 6388 raft_consensus.cc:504] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.300935 5840 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:59.301553 5838 ts_tablet_manager.cc:1888] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.302865 5840 tablet_bootstrap.cc:654] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.309612 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 190baa48859640cf928642dccbebfceb
I20240507 08:11:59.309834 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96ef77ef4f474107bf31395f5f30c12f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.311688 6377 tablet.cc:1722] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:11:59.312000 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a3e58ba8c26d4f17af24eb379aca1f60
I20240507 08:11:59.312872 6378 tablet.cc:1722] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:11:59.310865 6029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:11:59.313196 6029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
I20240507 08:11:59.313629 5837 ts_tablet_manager.cc:1909] T 6f6670e99e4a4442801d42950eb7ab79 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:59.314020 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6ad0c511f95b4b09abad3bc4a13d1238" } modify_external_catalogs: true
I20240507 08:11:59.314308 5257 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet a2e0811558ab41fda39e688356c19d69
I20240507 08:11:59.314481 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aca8c38841db45e38ae5d8d965a16d0e
I20240507 08:11:59.314591 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6ad0c511f95b4b09abad3bc4a13d1238" } modify_external_catalogs: true
I20240507 08:11:59.320549 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_659bd38568664e459c30bf2ad102ff78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.321084 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_659bd38568664e459c30bf2ad102ff78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.324151 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30e1980aaabd40d4a156b14a39680f90"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.324606 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30e1980aaabd40d4a156b14a39680f90 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.328766 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.329022 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet aca8c38841db45e38ae5d8d965a16d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.329201 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 190baa48859640cf928642dccbebfceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.331583 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet aca8c38841db45e38ae5d8d965a16d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.332233 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 6f6670e99e4a4442801d42950eb7ab79 (table default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]) successfully deleted
W20240507 08:11:59.332604 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet aca8c38841db45e38ae5d8d965a16d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aca8c38841db45e38ae5d8d965a16d0e already in progress: creating tablet
W20240507 08:11:59.332988 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 831153d2b89e43f8b08b086f7a8f3a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 831153d2b89e43f8b08b086f7a8f3a54 already in progress: creating tablet
W20240507 08:11:59.333273 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0d617213b959452ab6519809e56db128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d617213b959452ab6519809e56db128 already in progress: creating tablet
W20240507 08:11:59.333508 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 072702ef2a6d43248132da6aa3a2d493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072702ef2a6d43248132da6aa3a2d493 already in progress: creating tablet
W20240507 08:11:59.333995 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a3e58ba8c26d4f17af24eb379aca1f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e58ba8c26d4f17af24eb379aca1f60 already in progress: creating tablet
W20240507 08:11:59.334224 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet aca8c38841db45e38ae5d8d965a16d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aca8c38841db45e38ae5d8d965a16d0e already in progress: creating tablet
W20240507 08:11:59.334981 5520 leader_election.cc:341] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f922a7519084409db98693ee8797592b
I20240507 08:11:59.335870 5838 log.cc:1196] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/67c8624285d946e597227430a5714e2f
W20240507 08:11:59.336313 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 190baa48859640cf928642dccbebfceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190baa48859640cf928642dccbebfceb already in progress: creating tablet
I20240507 08:11:59.343119 5838 ts_tablet_manager.cc:1909] T 67c8624285d946e597227430a5714e2f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:11:59.347880 5517 leader_election.cc:341] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: f922a7519084409db98693ee8797592b
I20240507 08:11:59.349037 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2717b2d928674c17809ef4b0141e352e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.353667 5517 leader_election.cc:302] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:59.354041 6390 raft_consensus.cc:2726] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:59.354256 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 426) took 2040 ms. Trace:
I20240507 08:11:59.354368 5838 rpcz_store.cc:276] 0507 08:11:57.309798 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.309846 (+ 48us) service_pool.cc:225] Handling call
0507 08:11:58.608066 (+1298220us) ts_tablet_manager.cc:1120] Deleting tablet 67c8624285d946e597227430a5714e2f
0507 08:11:58.608073 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.214571 (+606498us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.301450 (+ 86879us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.350470 (+ 49020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1297975}
I20240507 08:11:59.356053 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.355079 6388 leader_election.cc:288] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:11:59.358860 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a3e58ba8c26d4f17af24eb379aca1f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e58ba8c26d4f17af24eb379aca1f60 already in progress: creating tablet
I20240507 08:11:59.359678 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8935f7c20ed43a69e9cfcb28fe5d023. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.360065 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 190baa48859640cf928642dccbebfceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:59.361016 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 190baa48859640cf928642dccbebfceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190baa48859640cf928642dccbebfceb already in progress: creating tablet
I20240507 08:11:59.369201 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a07279891494410096ed3d7da872b203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.369577 5838 tablet_replica.cc:330] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:59.371129 5260 catalog_manager.cc:5484] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:59.371639 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:11:59.371915 5260 catalog_manager.cc:5484] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:11:59.372352 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
W20240507 08:11:59.373642 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a07279891494410096ed3d7da872b203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.375144 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_287233b419e646b58b404eb7190a3ac6" } new_table_name: "default.Table_c29de2c9d7be43e29b930c0507562589" modify_external_catalogs: true
I20240507 08:11:59.375391 5838 raft_consensus.cc:2229] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.376466 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4230ce43f9b045a3a4fe583a727d0063" } new_table_name: "default.Table_0fafe65af5f34be4b333cb6baa327f94" modify_external_catalogs: true
I20240507 08:11:59.377701 5838 raft_consensus.cc:2258] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.378266 5838 ts_tablet_manager.cc:1875] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.380095 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb9a13f3f676414d8c95abcc5889b14e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.380753 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb9a13f3f676414d8c95abcc5889b14e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.386845 5417 tablet_service.cc:1467] Processing CreateTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f (DEFAULT_TABLE table=default.Table_6828726f0a3b436faf6ac20342edff29 [id=97ff496559474b24b15ab9935c4860dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.387353 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c8db73b8cb4212b59d40bd6864e23f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.391089 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6563c42d240b4e8685dfa4972f2b9924 consensus state has no leader
I20240507 08:11:59.393215 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 190baa48859640cf928642dccbebfceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.395097 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc23d117e7e46d894931712891ffd56 (DEFAULT_TABLE table=default.Table_cec30b6d6641468b8bb09f32c9162560 [id=abe6acb616924a29b16a1873748bd4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.395562 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc23d117e7e46d894931712891ffd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.396821 5843 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:59.397354 5839 tablet_bootstrap.cc:492] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:59.401679 5839 ts_tablet_manager.cc:1367] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.198s user 0.003s sys 0.003s
I20240507 08:11:59.402555 5839 raft_consensus.cc:348] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.412227 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet aca8c38841db45e38ae5d8d965a16d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.412470 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.416036 5839 raft_consensus.cc:374] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.421054 6378 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:59.421785 6376 raft_consensus.cc:482] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:59.421955 6376 raft_consensus.cc:504] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.422120 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc23d117e7e46d894931712891ffd56 (DEFAULT_TABLE table=default.Table_cec30b6d6641468b8bb09f32c9162560 [id=abe6acb616924a29b16a1873748bd4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.422590 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc23d117e7e46d894931712891ffd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.422662 6376 leader_election.cc:288] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:59.407830 6377 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:11:59.425331 5843 ts_tablet_manager.cc:1367] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.729s user 0.024s sys 0.033s
I20240507 08:11:59.427742 5691 tablet_service.cc:1467] Processing CreateTablet for tablet fd6933ac8ee3452db6edf3c255247b33 (DEFAULT_TABLE table=default.Table_61d23e54efcc415c9f29270b3e622bc0 [id=796abf6270aa4052baf2340453fa31e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.428260 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6933ac8ee3452db6edf3c255247b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.430910 5839 raft_consensus.cc:729] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.431493 5839 consensus_queue.cc:260] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.432075 5839 ts_tablet_manager.cc:1398] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240507 08:11:59.431962 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:11:59.435462 5654 leader_election.cc:341] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:59.440109 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 916e33e40ef44d06a359b82d8fe48157
I20240507 08:11:59.440915 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bd945626c197488da2bb9d0d8e8a55dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.441478 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd945626c197488da2bb9d0d8e8a55dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.442317 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:59.443068 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 916e33e40ef44d06a359b82d8fe48157 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 916e33e40ef44d06a359b82d8fe48157 already in progress: creating tablet
I20240507 08:11:59.443341 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_19b106706150440f8c07e95b72d7bde8" } new_table_name: "default.Table_dc3c2099e75b478ba03e7084ecd2e317" modify_external_catalogs: true
I20240507 08:11:59.445173 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 33a425a90f0449bcaaacb4c3f653d26c from {username='slave'} at 127.0.0.1:44738
W20240507 08:11:59.445942 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 310c601abead42e0bd2a2df1a0e16ec2 consensus state has no leader
I20240507 08:11:59.445833 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a894109d6af4424db5114275d258a4f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.446341 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a894109d6af4424db5114275d258a4f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.446277 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_630ea5aaa63747deaf53c4b3b58cf2be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.446748 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_630ea5aaa63747deaf53c4b3b58cf2be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:59.447862 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43225934cef04109b3dbc057b230f032 consensus state has no leader
I20240507 08:11:59.448196 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_12604f6607614bb6a9001e40b82cabe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.448671 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_12604f6607614bb6a9001e40b82cabe3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.450567 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:59.451218 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_2071d1f910a644788fb77006559bb1eb [id=2435a4b9c9a7464aafa19339bf7c9277] alter complete (version 4)
I20240507 08:11:59.456331 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2fc23d117e7e46d894931712891ffd56 (DEFAULT_TABLE table=default.Table_cec30b6d6641468b8bb09f32c9162560 [id=abe6acb616924a29b16a1873748bd4ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.462361 5839 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:59.469784 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fc23d117e7e46d894931712891ffd56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.472028 5563 tablet_service.cc:1467] Processing CreateTablet for tablet fd6933ac8ee3452db6edf3c255247b33 (DEFAULT_TABLE table=default.Table_61d23e54efcc415c9f29270b3e622bc0 [id=796abf6270aa4052baf2340453fa31e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.473511 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet f922a7519084409db98693ee8797592b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.475621 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_9735652cdc0d447080df766780402da6 [id=0a1bc06a351549d5aea075ad8d026476]). Expected version 3 got 0
I20240507 08:11:59.476290 5691 tablet_service.cc:1467] Processing CreateTablet for tablet ed0018356e254b82995061ef7290556d (DEFAULT_TABLE table=default.Table_139fa8ac0afe4140ac71e91821a98481 [id=5c27869179144a7eb2324c0f99661ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.476836 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0018356e254b82995061ef7290556d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.489277 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:11:59.491933 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5782cfa5eca74c19a2e724a9124597f2 consensus state has no leader
I20240507 08:11:59.493724 5548 tablet_service.cc:1467] Processing CreateTablet for tablet ed0018356e254b82995061ef7290556d (DEFAULT_TABLE table=default.Table_139fa8ac0afe4140ac71e91821a98481 [id=5c27869179144a7eb2324c0f99661ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.495599 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6933ac8ee3452db6edf3c255247b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.500527 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" candidate_term: 2 candidate_status { last_received { term: 1 index: 601 } } ignore_live_leader: false dest_uuid: "94c202eef7f3466fa3e703644b40879d" is_pre_election: true
I20240507 08:11:59.503898 6150 leader_election.cc:302] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a39de2a69ed4492aab4d7a232b09c1ae; no voters: 94c202eef7f3466fa3e703644b40879d, a91047a8a5414eff9c06ca9ca2b548e8
I20240507 08:11:59.506536 5839 tablet_bootstrap.cc:654] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.506804 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0018356e254b82995061ef7290556d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.514269 5840 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:59.514477 5840 ts_tablet_manager.cc:1367] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.214s user 0.056s sys 0.091s
I20240507 08:11:59.515885 5840 raft_consensus.cc:348] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:11:59.517742 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 310c601abead42e0bd2a2df1a0e16ec2 (table default.Table_c29de2c9d7be43e29b930c0507562589 [id=28572caf37234d87953c7b84a1bb70d9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 310c601abead42e0bd2a2df1a0e16ec2 consensus state has no leader
I20240507 08:11:59.530121 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b17576c2e21b405380ccce99ebbd7b7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.530712 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17576c2e21b405380ccce99ebbd7b7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.532738 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fdbd31dead3043cba3f6dce0b222bd13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.533211 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdbd31dead3043cba3f6dce0b222bd13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.518779 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 (DEFAULT_TABLE table=default.Table_3ad8e609d07646e49e01cc55ba8ee969 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.534024 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6563c42d240b4e8685dfa4972f2b9924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.534344 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_825e44ca78c14d6392c7e6864babde4c" } modify_external_catalogs: true
I20240507 08:11:59.534511 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_825e44ca78c14d6392c7e6864babde4c" } modify_external_catalogs: true
I20240507 08:11:59.535753 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6828726f0a3b436faf6ac20342edff29" } new_table_name: "default.Table_d035b6d966af4ae1921685ae1bd3337a" modify_external_catalogs: true
I20240507 08:11:59.537184 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ffbb2e35cc34f288580fefe1e79ebfd" } modify_external_catalogs: true
I20240507 08:11:59.537357 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ffbb2e35cc34f288580fefe1e79ebfd" } modify_external_catalogs: true
I20240507 08:11:59.529857 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a (DEFAULT_TABLE table=default.Table_67202ebc0080457d8ccacb243261511e [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.538400 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1eeccee8e49b3a1dbd1930c2d108a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.538913 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f993482dcb444626a4d006bf1755dcca" } modify_external_catalogs: true
I20240507 08:11:59.539055 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f993482dcb444626a4d006bf1755dcca" } modify_external_catalogs: true
I20240507 08:11:59.541117 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4ad9488db7b43469ea4ec3d4e365ba4" } modify_external_catalogs: true
I20240507 08:11:59.541316 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4ad9488db7b43469ea4ec3d4e365ba4" } modify_external_catalogs: true
I20240507 08:11:59.541323 5838 ts_tablet_manager.cc:1888] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.530499 5698 tablet_service.cc:1467] Processing CreateTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.542055 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.549070 5838 log.cc:1196] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/344bf4ab39f74accb383d695578de604
I20240507 08:11:59.516108 5840 raft_consensus.cc:374] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.555815 5840 raft_consensus.cc:729] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.557008 5840 consensus_queue.cc:260] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.566498 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet a5976eca006448e0b9f2573acd738bb1 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.568367 5840 ts_tablet_manager.cc:1398] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.054s user 0.000s sys 0.003s
I20240507 08:11:59.569213 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:59.570516 5416 tablet_service.cc:1467] Processing CreateTablet for tablet fd6933ac8ee3452db6edf3c255247b33 (DEFAULT_TABLE table=default.Table_61d23e54efcc415c9f29270b3e622bc0 [id=796abf6270aa4052baf2340453fa31e5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:59.569945 5656 leader_election.cc:341] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:11:59.570451 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 456eb5e63eb5435f9f60e351478bc871 (DEFAULT_TABLE table=default.Table_ad303c5560534112a84810140d744045 [id=cbfda8d84a094f6490d09c354bd45ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.572247 5840 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:59.572600 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456eb5e63eb5435f9f60e351478bc871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.572336 5656 leader_election.cc:302] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:11:59.574306 6376 raft_consensus.cc:2726] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:11:59.581550 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 76f25517f0b84fc4ac4462ed8a15854d (DEFAULT_TABLE table=default.Table_edfefd7112b84facba5e8f004b5c1c1c [id=5b6a3fba662e4544b0890caa29f70199]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.582110 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f25517f0b84fc4ac4462ed8a15854d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.585713 6378 tablet.cc:1717] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Skipping requested alter to schema version 3, tablet already version 3
I20240507 08:11:59.589896 6377 tablet.cc:1717] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Skipping requested alter to schema version 3, tablet already version 3
I20240507 08:11:59.591372 5840 tablet_bootstrap.cc:654] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.598121 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd6933ac8ee3452db6edf3c255247b33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.599903 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a2b806f32a0d49a69d6eb9eb5f235f52" } new_table_name: "default.Table_f702440833504e4eb59747ee6de0cefb" modify_external_catalogs: true
I20240507 08:11:59.603801 6235 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" instance_seqno: 1715069508741182) as {username='slave'} at 127.4.246.194:49123; Asking this server to re-register.
I20240507 08:11:59.604387 6383 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:11:59.608392 6237 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" instance_seqno: 1715069508237331) as {username='slave'} at 127.4.246.193:51001; Asking this server to re-register.
I20240507 08:11:59.609750 5494 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:59.615105 5843 raft_consensus.cc:348] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.615489 5843 raft_consensus.cc:374] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.615751 5843 raft_consensus.cc:729] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.616703 5843 consensus_queue.cc:260] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.620215 5632 heartbeater.cc:464] Registering TS with master...
I20240507 08:11:59.624332 5838 ts_tablet_manager.cc:1909] T 344bf4ab39f74accb383d695578de604 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:59.624698 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 427) took 2301 ms. Trace:
I20240507 08:11:59.625876 5838 rpcz_store.cc:276] 0507 08:11:57.323473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.323630 (+ 157us) service_pool.cc:225] Handling call
0507 08:11:59.354659 (+2031029us) ts_tablet_manager.cc:1120] Deleting tablet 344bf4ab39f74accb383d695578de604
0507 08:11:59.356933 (+ 2274us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.495392 (+138459us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.532384 (+ 36992us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.624658 (+ 92274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2030757}
I20240507 08:11:59.626295 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 434) took 1939 ms. Trace:
I20240507 08:11:59.627156 5838 rpcz_store.cc:276] 0507 08:11:57.686578 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.686686 (+ 108us) service_pool.cc:225] Handling call
0507 08:11:59.626163 (+1939477us) ts_tablet_manager.cc:1120] Deleting tablet 47a154769d0f4501837888feba98f33d
0507 08:11:59.626171 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.626267 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1935915}
I20240507 08:11:59.627566 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 437) took 1925 ms. Trace:
I20240507 08:11:59.633636 5838 rpcz_store.cc:276] 0507 08:11:57.702174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.702655 (+ 481us) service_pool.cc:225] Handling call
0507 08:11:59.627437 (+1924782us) ts_tablet_manager.cc:1120] Deleting tablet e14487e625374060b118c9cce4500b3f
0507 08:11:59.627449 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.627535 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1861130}
I20240507 08:11:59.634020 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 448) took 1850 ms. Trace:
I20240507 08:11:59.634279 6237 ts_manager.cc:194] Registered new tserver with Master: dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:11:59.636406 6238 ts_manager.cc:194] Registered new tserver with Master: fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:11:59.641990 5261 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_9735652cdc0d447080df766780402da6 [id=0a1bc06a351549d5aea075ad8d026476] alter complete (version 3)
I20240507 08:11:59.644923 5838 rpcz_store.cc:276] 0507 08:11:57.783737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.784488 (+ 751us) service_pool.cc:225] Handling call
0507 08:11:59.633895 (+1849407us) ts_tablet_manager.cc:1120] Deleting tablet 7d89e4185229431f9a68f599da4aff34
0507 08:11:59.633903 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.633990 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1845925}
I20240507 08:11:59.645479 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 447) took 1861 ms. Trace:
I20240507 08:11:59.646103 5838 rpcz_store.cc:276] 0507 08:11:57.783635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.784457 (+ 822us) service_pool.cc:225] Handling call
0507 08:11:59.645321 (+1860864us) ts_tablet_manager.cc:1120] Deleting tablet 565ac6091f5e48b1b024094d5560e275
0507 08:11:59.645330 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.645450 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1856430}
I20240507 08:11:59.646605 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 454) took 1761 ms. Trace:
I20240507 08:11:59.646956 5838 rpcz_store.cc:276] 0507 08:11:57.884908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.885126 (+ 218us) service_pool.cc:225] Handling call
0507 08:11:59.646465 (+1761339us) ts_tablet_manager.cc:1120] Deleting tablet 049483b0533642f2b46d8fa0e9b123cc
0507 08:11:59.646474 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.646567 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742282}
I20240507 08:11:59.647385 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 453) took 1807 ms. Trace:
I20240507 08:11:59.647742 5838 rpcz_store.cc:276] 0507 08:11:57.878350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:57.888287 (+ 9937us) service_pool.cc:225] Handling call
0507 08:11:59.647274 (+1758987us) ts_tablet_manager.cc:1120] Deleting tablet 3a94c97ecc5d47abb190f3833fd317ca
0507 08:11:59.647286 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.647354 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1739948}
I20240507 08:11:59.648291 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 455) took 1410 ms. Trace:
I20240507 08:11:59.648535 5838 rpcz_store.cc:276] 0507 08:11:58.238190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.238251 (+ 61us) service_pool.cc:225] Handling call
0507 08:11:59.648143 (+1409892us) ts_tablet_manager.cc:1120] Deleting tablet eec3704b135740f1a906d60f3ea87ffe
0507 08:11:59.648151 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.648217 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409303}
I20240507 08:11:59.649513 5838 tablet_replica.cc:330] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:59.653079 5838 raft_consensus.cc:2229] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.653343 5838 raft_consensus.cc:2258] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.659790 5838 ts_tablet_manager.cc:1875] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.660024 5839 tablet_bootstrap.cc:492] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:11:59.661334 5839 ts_tablet_manager.cc:1367] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.221s user 0.001s sys 0.005s
I20240507 08:11:59.663094 5843 ts_tablet_manager.cc:1398] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.235s user 0.001s sys 0.002s
I20240507 08:11:59.663414 5843 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:59.664933 5843 tablet_bootstrap.cc:654] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:59.667703 5249 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 157802us
I20240507 08:11:59.668406 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 344bf4ab39f74accb383d695578de604 (table default.Table_e80b1599b03348dfa18d151527ece682 [id=b1c8d1d7d2654336a7029c38a0284824]) successfully deleted
W20240507 08:11:59.668640 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 47a154769d0f4501837888feba98f33d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 47a154769d0f4501837888feba98f33d already in progress: creating tablet
W20240507 08:11:59.668815 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e14487e625374060b118c9cce4500b3f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e14487e625374060b118c9cce4500b3f already in progress: creating tablet
W20240507 08:11:59.668995 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7d89e4185229431f9a68f599da4aff34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d89e4185229431f9a68f599da4aff34 already in progress: creating tablet
W20240507 08:11:59.669219 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 565ac6091f5e48b1b024094d5560e275 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 565ac6091f5e48b1b024094d5560e275 already in progress: creating tablet
W20240507 08:11:59.669443 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 049483b0533642f2b46d8fa0e9b123cc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 049483b0533642f2b46d8fa0e9b123cc already in progress: creating tablet
W20240507 08:11:59.669665 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3a94c97ecc5d47abb190f3833fd317ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a94c97ecc5d47abb190f3833fd317ca already in progress: creating tablet
I20240507 08:11:59.669739 5691 tablet_service.cc:1467] Processing CreateTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a (DEFAULT_TABLE table=default.Table_94c6194476954481ab568218481b03c8 [id=05c53ba6185b4b06b0fcdd5226a5db18]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:11:59.669925 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet eec3704b135740f1a906d60f3ea87ffe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eec3704b135740f1a906d60f3ea87ffe already in progress: creating tablet
W20240507 08:11:59.670742 5790 log.cc:924] Time spent T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Append to log took a long time: real 0.160s user 0.000s sys 0.000s
I20240507 08:11:59.671455 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a3ccd7a5e4b9e956735443d4c484a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.671757 5563 tablet_service.cc:1467] Processing CreateTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.672151 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.674111 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d73950d9f420413693022530757e7ead" } modify_external_catalogs: true
I20240507 08:11:59.674288 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d73950d9f420413693022530757e7ead" } modify_external_catalogs: true
I20240507 08:11:59.675534 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a (DEFAULT_TABLE table=default.Table_67202ebc0080457d8ccacb243261511e [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.675905 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1eeccee8e49b3a1dbd1930c2d108a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.677814 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 456eb5e63eb5435f9f60e351478bc871 (DEFAULT_TABLE table=default.Table_ad303c5560534112a84810140d744045 [id=cbfda8d84a094f6490d09c354bd45ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.678165 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456eb5e63eb5435f9f60e351478bc871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.679929 5840 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:59.680095 5840 ts_tablet_manager.cc:1367] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.108s user 0.008s sys 0.008s
I20240507 08:11:59.680827 5840 raft_consensus.cc:348] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.681075 5840 raft_consensus.cc:374] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.681164 5840 raft_consensus.cc:729] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.681445 5840 consensus_queue.cc:260] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.681840 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 76f25517f0b84fc4ac4462ed8a15854d (DEFAULT_TABLE table=default.Table_edfefd7112b84facba5e8f004b5c1c1c [id=5b6a3fba662e4544b0890caa29f70199]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.682176 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f25517f0b84fc4ac4462ed8a15854d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.685045 5840 ts_tablet_manager.cc:1398] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:11:59.687520 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a5976eca006448e0b9f2573acd738bb1
I20240507 08:11:59.687995 5258 catalog_manager.cc:6554] ReplaceTablet: tablet a5976eca006448e0b9f2573acd738bb1 deleted and replaced by tablet 5481ddbcd31449c4b44afb87a97138bd
I20240507 08:11:59.688385 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a9eff56d2c9c422a94059a8501a04469"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.688930 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9eff56d2c9c422a94059a8501a04469 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.689183 5840 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:59.689651 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet a5976eca006448e0b9f2573acd738bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 5481ddbcd31449c4b44afb87a97138bd) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:59.690624 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a5976eca006448e0b9f2573acd738bb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5976eca006448e0b9f2573acd738bb1 already in progress: creating tablet
I20240507 08:11:59.692044 5261 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c] alter complete (version 1)
W20240507 08:11:59.692504 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1c8db73b8cb4212b59d40bd6864e23f consensus state has no leader
I20240507 08:11:59.692651 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5712fb27d6da4fdb995d7e583f33f616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.692878 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c0775c295b804c7e987bafb2f8b19bed" } new_table_name: "default.Table_1cb491139e6446b19eef66a5838468a7" modify_external_catalogs: true
I20240507 08:11:59.693177 5840 tablet_bootstrap.cc:654] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:11:59.693168 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5712fb27d6da4fdb995d7e583f33f616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.694852 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0fb75933874a948281bf573a1723a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.696069 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6b4c69d38e484f5c819b7dd915bf4410
I20240507 08:11:59.696794 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c29de2c9d7be43e29b930c0507562589" } new_table_name: "default.Table_638c97cc24874e01afa74a619d4b9dc4" modify_external_catalogs: true
I20240507 08:11:59.698698 5837 tablet_replica.cc:330] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:59.698920 5837 raft_consensus.cc:2229] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.699087 5837 raft_consensus.cc:2258] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.699741 5837 ts_tablet_manager.cc:1875] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.702828 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e3df44d5188247b6a317ba4185f0b745"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.703315 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3df44d5188247b6a317ba4185f0b745 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.705163 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.706307 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e61891ba7bda49f99a67145edfe93e5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.706743 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e61891ba7bda49f99a67145edfe93e5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:59.708103 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 645ae26bb26f43d298afe98817d88fa4 consensus state has no leader
I20240507 08:11:59.709009 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b31fdadee60e4f5e94e92c536e18774d
I20240507 08:11:59.709156 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 156bf04b57724134a45fa9fc477596c5
I20240507 08:11:59.710907 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f513573e340547d6ae5ab848d00ace72
I20240507 08:11:59.718750 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 (DEFAULT_TABLE table=default.Table_3ad8e609d07646e49e01cc55ba8ee969 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.719215 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6563c42d240b4e8685dfa4972f2b9924. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.724185 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 310c601abead42e0bd2a2df1a0e16ec2 consensus state has no leader
I20240507 08:11:59.733928 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1f43a80d38ed4788a13421b245fe6e8b
I20240507 08:11:59.739454 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.740331 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 156bf04b57724134a45fa9fc477596c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.740592 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet b31fdadee60e4f5e94e92c536e18774d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.740974 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet f513573e340547d6ae5ab848d00ace72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.741346 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.743402 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet b31fdadee60e4f5e94e92c536e18774d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.750766 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6f66b9533a8d40b093d99097ef61af87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.751559 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f66b9533a8d40b093d99097ef61af87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.751612 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8881f6c0a43c45f6b13ffc2d61012180" } new_table_name: "default.Table_a314de990d004c20810e4ce3cc8c22dd" modify_external_catalogs: true
I20240507 08:11:59.753078 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3634efacf6494a4eabe5a4d9451e64a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.753250 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ced552bcfdd14427a0b6990736d3165c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.753535 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3634efacf6494a4eabe5a4d9451e64a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:11:59.753934 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced552bcfdd14427a0b6990736d3165c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.754923 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_65b762c4d6ac480eb6fbc20827e51a8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.755302 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65b762c4d6ac480eb6fbc20827e51a8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.755831 6388 raft_consensus.cc:482] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:59.755990 6388 raft_consensus.cc:504] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.756687 6388 leader_election.cc:288] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:59.764602 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet a5976eca006448e0b9f2573acd738bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 5481ddbcd31449c4b44afb87a97138bd) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.765756 5553 tablet_service.cc:1467] Processing CreateTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a (DEFAULT_TABLE table=default.Table_94c6194476954481ab568218481b03c8 [id=05c53ba6185b4b06b0fcdd5226a5db18]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.765895 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:11:59.766158 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a3ccd7a5e4b9e956735443d4c484a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.770576 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c4cd7b56d4e468eb754d9653117d023 consensus state has no leader
I20240507 08:11:59.771353 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.772106 5421 tablet_service.cc:1467] Processing CreateTablet for tablet ed0018356e254b82995061ef7290556d (DEFAULT_TABLE table=default.Table_139fa8ac0afe4140ac71e91821a98481 [id=5c27869179144a7eb2324c0f99661ff0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.772511 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed0018356e254b82995061ef7290556d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.780015 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 (DEFAULT_TABLE table=default.Table_3ad8e609d07646e49e01cc55ba8ee969 [id=c5ee2823c21745f380337f43009224f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.780050 5417 tablet_service.cc:1467] Processing CreateTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 (DEFAULT_TABLE table=default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.780468 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.786556 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08aac6e88b7c41d8a59d1e12e7f6c14e consensus state has no leader
I20240507 08:11:59.789175 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6563c42d240b4e8685dfa4972f2b9924. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.668965 5839 raft_consensus.cc:348] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.790634 5839 raft_consensus.cc:374] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.789860 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a (DEFAULT_TABLE table=default.Table_67202ebc0080457d8ccacb243261511e [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.790475 5417 tablet_service.cc:1467] Processing CreateTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a (DEFAULT_TABLE table=default.Table_94c6194476954481ab568218481b03c8 [id=05c53ba6185b4b06b0fcdd5226a5db18]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.791659 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68f1eeccee8e49b3a1dbd1930c2d108a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.798288 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d33a3ccd7a5e4b9e956735443d4c484a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.798352 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 1f5d566695d94c708e90ee30254084a1 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.791447 5839 raft_consensus.cc:729] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.802150 5839 consensus_queue.cc:260] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.807629 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 270a1fd15925416483c211021549315c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.808187 5839 ts_tablet_manager.cc:1398] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.146s user 0.001s sys 0.001s
I20240507 08:11:59.808414 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 456eb5e63eb5435f9f60e351478bc871 (DEFAULT_TABLE table=default.Table_ad303c5560534112a84810140d744045 [id=cbfda8d84a094f6490d09c354bd45ef8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.808815 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 456eb5e63eb5435f9f60e351478bc871. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.812475 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet b2df872d04b94c30872b03367df3cc31 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.813165 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 193ec0b463d04835b74df80229c85cde from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.813977 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 156bf04b57724134a45fa9fc477596c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.814121 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1f5d566695d94c708e90ee30254084a1
I20240507 08:11:59.814267 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1c6b488b9014841899597d858bda5e2" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:11:59.814749 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 1f5d566695d94c708e90ee30254084a1 deleted and replaced by tablet 12e85df8b900468ba2584fd8f2e9bb2d
I20240507 08:11:59.815722 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.817107 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5d566695d94c708e90ee30254084a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 12e85df8b900468ba2584fd8f2e9bb2d) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.817601 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.817740 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet f513573e340547d6ae5ab848d00ace72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.817859 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.813681 6388 raft_consensus.cc:482] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:59.819216 6388 raft_consensus.cc:504] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.819900 6388 leader_election.cc:288] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:59.814785 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5d566695d94c708e90ee30254084a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 12e85df8b900468ba2584fd8f2e9bb2d) from {username='slave'} at 127.0.0.1:37096
I20240507 08:11:59.815004 5517 leader_election.cc:302] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:59.820556 6388 raft_consensus.cc:2726] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:59.823788 5517 leader_election.cc:341] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 270a1fd15925416483c211021549315c
I20240507 08:11:59.826751 5837 ts_tablet_manager.cc:1888] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:11:59.827275 5837 log.cc:1196] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5e0fb75933874a948281bf573a1723a1
I20240507 08:11:59.827363 5839 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:11:59.828733 5837 ts_tablet_manager.cc:1909] T 5e0fb75933874a948281bf573a1723a1 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:11:59.829072 5839 tablet_bootstrap.cc:654] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.835093 5840 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:11:59.835247 5840 ts_tablet_manager.cc:1367] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.146s user 0.014s sys 0.017s
I20240507 08:11:59.835513 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet a5976eca006448e0b9f2573acd738bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 5481ddbcd31449c4b44afb87a97138bd) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.836869 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5e0fb75933874a948281bf573a1723a1 (table default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]) successfully deleted
I20240507 08:11:59.836834 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 76f25517f0b84fc4ac4462ed8a15854d (DEFAULT_TABLE table=default.Table_edfefd7112b84facba5e8f004b5c1c1c [id=5b6a3fba662e4544b0890caa29f70199]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.837234 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76f25517f0b84fc4ac4462ed8a15854d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.841537 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 156bf04b57724134a45fa9fc477596c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.841802 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet f513573e340547d6ae5ab848d00ace72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.841807 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:11:59.842463 5520 leader_election.cc:341] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 270a1fd15925416483c211021549315c
I20240507 08:11:59.842624 5520 leader_election.cc:302] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:11:59.843142 6388 raft_consensus.cc:2726] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:11:59.849352 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 916e33e40ef44d06a359b82d8fe48157 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 916e33e40ef44d06a359b82d8fe48157 already in progress: creating tablet
W20240507 08:11:59.851517 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a5976eca006448e0b9f2573acd738bb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5976eca006448e0b9f2573acd738bb1 already in progress: creating tablet
W20240507 08:11:59.851929 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 156bf04b57724134a45fa9fc477596c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156bf04b57724134a45fa9fc477596c5 already in progress: creating tablet
W20240507 08:11:59.852178 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f513573e340547d6ae5ab848d00ace72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f513573e340547d6ae5ab848d00ace72 already in progress: creating tablet
W20240507 08:11:59.852378 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1f43a80d38ed4788a13421b245fe6e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f43a80d38ed4788a13421b245fe6e8b already in progress: creating tablet
I20240507 08:11:59.857434 5843 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:11:59.862484 5840 raft_consensus.cc:348] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.862735 5840 raft_consensus.cc:374] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.862843 5840 raft_consensus.cc:729] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.863112 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet b31fdadee60e4f5e94e92c536e18774d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.863094 5840 consensus_queue.cc:260] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.863394 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5d566695d94c708e90ee30254084a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 12e85df8b900468ba2584fd8f2e9bb2d) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.863668 5840 ts_tablet_manager.cc:1398] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.028s user 0.001s sys 0.001s
I20240507 08:11:59.863903 5841 tablet_replica.cc:330] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:59.864028 5840 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:11:59.864080 5841 raft_consensus.cc:2229] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:11:59.864425 5841 raft_consensus.cc:2258] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.868189 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:55948
W20240507 08:11:59.868217 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b31fdadee60e4f5e94e92c536e18774d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b31fdadee60e4f5e94e92c536e18774d already in progress: creating tablet
I20240507 08:11:59.868273 5841 ts_tablet_manager.cc:1875] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.868726 5840 tablet_bootstrap.cc:654] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.873675 5839 tablet_bootstrap.cc:492] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:11:59.873756 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6b4c69d38e484f5c819b7dd915bf4410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4c69d38e484f5c819b7dd915bf4410 already in progress: creating tablet
I20240507 08:11:59.874321 5839 ts_tablet_manager.cc:1367] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.002s
W20240507 08:11:59.874580 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b31fdadee60e4f5e94e92c536e18774d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b31fdadee60e4f5e94e92c536e18774d already in progress: creating tablet
W20240507 08:11:59.875313 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 156bf04b57724134a45fa9fc477596c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156bf04b57724134a45fa9fc477596c5 already in progress: creating tablet
I20240507 08:11:59.875356 5839 raft_consensus.cc:348] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.875658 5839 raft_consensus.cc:374] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.875690 5837 tablet_replica.cc:330] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:11:59.875753 5839 raft_consensus.cc:729] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.876255 5837 raft_consensus.cc:2229] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.876494 5837 raft_consensus.cc:2258] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.876762 5843 ts_tablet_manager.cc:1367] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.196s user 0.001s sys 0.010s
I20240507 08:11:59.877311 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet d0ea74702c4b4514a31f0fad7e7a2f5b (table default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972]). Expected version 2 got 0
I20240507 08:11:59.878084 5837 ts_tablet_manager.cc:1875] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.880400 5843 raft_consensus.cc:348] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.885659 5843 raft_consensus.cc:374] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:11:59.885799 5843 raft_consensus.cc:729] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:11:59.886196 5843 consensus_queue.cc:260] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:11:59.886965 5843 ts_tablet_manager.cc:1398] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:11:59.889079 5841 ts_tablet_manager.cc:1888] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:11:59.889184 5841 log.cc:1196] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1f5d566695d94c708e90ee30254084a1
I20240507 08:11:59.889673 5841 ts_tablet_manager.cc:1909] T 1f5d566695d94c708e90ee30254084a1 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:11:59.889981 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.892354 5838 ts_tablet_manager.cc:1888] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:11:59.892520 5838 log.cc:1196] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5405f264bac1464a8f1b4435715a64a4
I20240507 08:11:59.892967 5838 ts_tablet_manager.cc:1909] T 5405f264bac1464a8f1b4435715a64a4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:11:59.893985 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5ee780b3d01a481b80949a3ade7e3cd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.894812 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ee780b3d01a481b80949a3ade7e3cd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.895565 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2f9027dff4a546ffb9d51a689687e01c" } new_table_name: "default.Table_ad2e433251d046b589865483014c790e" modify_external_catalogs: true
I20240507 08:11:59.896137 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.899101 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d035b6d966af4ae1921685ae1bd3337a" } modify_external_catalogs: true
I20240507 08:11:59.899274 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d035b6d966af4ae1921685ae1bd3337a" } modify_external_catalogs: true
I20240507 08:11:59.899029 5839 consensus_queue.cc:260] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.900645 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 456) took 1659 ms. Trace:
I20240507 08:11:59.900753 5838 rpcz_store.cc:276] 0507 08:11:58.240937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.240992 (+ 55us) service_pool.cc:225] Handling call
0507 08:11:59.649444 (+1408452us) ts_tablet_manager.cc:1120] Deleting tablet 5405f264bac1464a8f1b4435715a64a4
0507 08:11:59.649453 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.842236 (+192783us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.892214 (+ 49978us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.900584 (+ 8370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1408219}
I20240507 08:11:59.901166 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5405f264bac1464a8f1b4435715a64a4 (table default.Table_55f22a879f524a07ba3a6edf9b32d822 [id=084d6a96fa14463c9dc4c8b48c92eeaa]) successfully deleted
I20240507 08:11:59.901213 5557 tablet_service.cc:1467] Processing CreateTablet for tablet e182d186e6fe4e388a68c61ccf159f8d (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.901865 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 (DEFAULT_TABLE table=default.Table_30e1980aaabd40d4a156b14a39680f90 [id=179c383e27e04b3485200df7f2555189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.902072 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e182d186e6fe4e388a68c61ccf159f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.902117 5838 tablet_replica.cc:330] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:11:59.904018 5838 raft_consensus.cc:2229] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.905800 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb49743134b4f84aa6fdd9760c13504. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.908780 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3078f2bf97d42d4bde80035e04a47a6 consensus state has no leader
I20240507 08:11:59.910316 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 03a59c7257e74b4abaa4ea7a1ee9136d from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.910473 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d1c8db73b8cb4212b59d40bd6864e23f
I20240507 08:11:59.920984 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:11:59.921594 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 03a59c7257e74b4abaa4ea7a1ee9136d deleted and replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb
I20240507 08:11:59.922432 5838 raft_consensus.cc:2258] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.922485 6388 raft_consensus.cc:482] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:11:59.922654 6388 raft_consensus.cc:504] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:11:59.923028 5838 ts_tablet_manager.cc:1875] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.923364 6388 leader_election.cc:288] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:11:59.932439 5843 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:11:59.934124 5691 tablet_service.cc:1467] Processing CreateTablet for tablet e182d186e6fe4e388a68c61ccf159f8d (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.935002 5839 ts_tablet_manager.cc:1398] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.061s user 0.003s sys 0.000s
I20240507 08:11:59.935501 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e182d186e6fe4e388a68c61ccf159f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.937796 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 (DEFAULT_TABLE table=default.Table_30e1980aaabd40d4a156b14a39680f90 [id=179c383e27e04b3485200df7f2555189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.938416 5415 tablet_service.cc:1467] Processing CreateTablet for tablet e182d186e6fe4e388a68c61ccf159f8d (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.939888 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.939990 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1f5d566695d94c708e90ee30254084a1 (table default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]) successfully deleted
I20240507 08:11:59.942247 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:11:59.942883 5843 tablet_bootstrap.cc:654] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:11:59.946136 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 (DEFAULT_TABLE table=default.Table_30e1980aaabd40d4a156b14a39680f90 [id=179c383e27e04b3485200df7f2555189]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:11:59.946517 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb49743134b4f84aa6fdd9760c13504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.946652 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5712fb27d6da4fdb995d7e583f33f616" } modify_external_catalogs: true
I20240507 08:11:59.946820 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5712fb27d6da4fdb995d7e583f33f616" } modify_external_catalogs: true
I20240507 08:11:59.947405 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8e9ea4672e9e4edcab31a6d54cd933e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.947942 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e9ea4672e9e4edcab31a6d54cd933e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.948756 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12317138695c42d695226bd546e857ab" } new_table_name: "default.Table_e4cc0dc4d852467c9bdeaeb440199da8" modify_external_catalogs: true
I20240507 08:11:59.949414 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_15f91f4b2cc24f138a6758b2a182bee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.950372 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a9eff56d2c9c422a94059a8501a04469" } modify_external_catalogs: true
I20240507 08:11:59.950537 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a9eff56d2c9c422a94059a8501a04469" } modify_external_catalogs: true
I20240507 08:11:59.951041 5841 tablet_replica.cc:330] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:11:59.951257 5841 raft_consensus.cc:2229] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:11:59.951359 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3eb49743134b4f84aa6fdd9760c13504. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:11:59.951438 5841 raft_consensus.cc:2258] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:11:59.952049 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.952225 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:11:59.956336 5841 ts_tablet_manager.cc:1875] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:11:59.961179 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fc64369fe72742ccb59a630ee95a8007" } modify_external_catalogs: true
I20240507 08:11:59.961369 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fc64369fe72742ccb59a630ee95a8007" } modify_external_catalogs: true
W20240507 08:11:59.961839 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15f91f4b2cc24f138a6758b2a182bee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.962620 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e85ccc16434742ac812466c315098b9a" } modify_external_catalogs: true
I20240507 08:11:59.962316 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_63548a17f66844648a2d71fb0c7d7175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:11:59.962759 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_96bb14500b6740f7bc08506e19f12519"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.964833 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63548a17f66844648a2d71fb0c7d7175 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.974364 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e182d186e6fe4e388a68c61ccf159f8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:11:59.975577 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96bb14500b6740f7bc08506e19f12519 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.963305 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e85ccc16434742ac812466c315098b9a" } modify_external_catalogs: true
I20240507 08:11:59.976428 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_953eb3a1426a4a479cb256cc68e6d4be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.984668 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_953eb3a1426a4a479cb256cc68e6d4be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.985131 5264 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 9c16b151c3b4491ebcd1bc7cc06fa77a
I20240507 08:11:59.986513 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_67202ebc0080457d8ccacb243261511e" } new_table_name: "default.Table_e860ab852fe14a5993248381d5e7e17d" modify_external_catalogs: true
W20240507 08:11:59.992264 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d4fac82e12d0437e950d01dbc4117efa consensus state has no leader
I20240507 08:11:59.992530 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 916e33e40ef44d06a359b82d8fe48157 from {username='slave'} at 127.0.0.1:44738
I20240507 08:11:59.993189 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_981d958d70ea464e8dd9a203a2374687"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:11:59.994421 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_981d958d70ea464e8dd9a203a2374687 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:11:59.999146 5837 ts_tablet_manager.cc:1888] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:11:59.939143 5520 leader_election.cc:341] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:11:59.951090 5838 ts_tablet_manager.cc:1888] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:12:00.005999 5520 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 66836.9us
I20240507 08:12:00.006112 5838 log.cc:1196] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/972bc0a5ba924e0fb946cdacd9575c1e
I20240507 08:12:00.006551 5838 ts_tablet_manager.cc:1909] T 972bc0a5ba924e0fb946cdacd9575c1e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.007396 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 457) took 1649 ms. Trace:
I20240507 08:12:00.007509 5838 rpcz_store.cc:276] 0507 08:11:58.358265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.359909 (+ 1644us) service_pool.cc:225] Handling call
0507 08:11:59.902015 (+1542106us) ts_tablet_manager.cc:1120] Deleting tablet 972bc0a5ba924e0fb946cdacd9575c1e
0507 08:11:59.902026 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:11:59.929199 (+ 27173us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:59.950956 (+ 21757us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:00.007359 (+ 56403us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18163,"tablet-delete.queue_time_us":1541813}
I20240507 08:12:00.007923 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 470) took 1453 ms. Trace:
I20240507 08:12:00.008044 5838 rpcz_store.cc:276] 0507 08:11:58.554907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.554975 (+ 68us) service_pool.cc:225] Handling call
0507 08:12:00.007783 (+1452808us) ts_tablet_manager.cc:1120] Deleting tablet c84a2018f1c442e2a4dfc1c6b9c5ceef
0507 08:12:00.007792 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.007895 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1451040}
I20240507 08:12:00.008597 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 469) took 1453 ms. Trace:
I20240507 08:12:00.008714 5838 rpcz_store.cc:276] 0507 08:11:58.554808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.556203 (+ 1395us) service_pool.cc:225] Handling call
0507 08:12:00.008267 (+1452064us) ts_tablet_manager.cc:1120] Deleting tablet 63ccdad494794192b3c37c8d5cbfcb17
0507 08:12:00.008464 (+ 197us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.008561 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446798}
I20240507 08:12:00.009110 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 467) took 1454 ms. Trace:
I20240507 08:12:00.009266 5838 rpcz_store.cc:276] 0507 08:11:58.554611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.556260 (+ 1649us) service_pool.cc:225] Handling call
0507 08:12:00.008939 (+1452679us) ts_tablet_manager.cc:1120] Deleting tablet 53fb38b567c5455ebc441e5f71db694f
0507 08:12:00.008951 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.009073 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1447227}
I20240507 08:12:00.009614 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 468) took 1454 ms. Trace:
I20240507 08:12:00.009722 5838 rpcz_store.cc:276] 0507 08:11:58.554707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.556236 (+ 1529us) service_pool.cc:225] Handling call
0507 08:12:00.009489 (+1453253us) ts_tablet_manager.cc:1120] Deleting tablet 5e0fb75933874a948281bf573a1723a1
0507 08:12:00.009497 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.009568 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1445396}
I20240507 08:12:00.010082 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 472) took 1409 ms. Trace:
I20240507 08:12:00.010222 5838 rpcz_store.cc:276] 0507 08:11:58.600316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.600382 (+ 66us) service_pool.cc:225] Handling call
0507 08:12:00.009932 (+1409550us) ts_tablet_manager.cc:1120] Deleting tablet 167ee9cbc9174b82824c468e098472a6
0507 08:12:00.009943 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.010048 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1409300}
I20240507 08:12:00.010622 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 487) took 1308 ms. Trace:
I20240507 08:12:00.010720 5838 rpcz_store.cc:276] 0507 08:11:58.702167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.705382 (+ 3215us) service_pool.cc:225] Handling call
0507 08:12:00.010492 (+1305110us) ts_tablet_manager.cc:1120] Deleting tablet 4e5439844f384513b4719cfe4f4e5d9b
0507 08:12:00.010503 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.010589 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1275111}
I20240507 08:12:00.011123 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 493) took 1109 ms. Trace:
I20240507 08:12:00.011248 5838 rpcz_store.cc:276] 0507 08:11:58.902028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.902415 (+ 387us) service_pool.cc:225] Handling call
0507 08:12:00.010948 (+1108533us) ts_tablet_manager.cc:1120] Deleting tablet 0d617213b959452ab6519809e56db128
0507 08:12:00.010958 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.011086 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1108304}
I20240507 08:12:00.011646 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 491) took 1109 ms. Trace:
I20240507 08:12:00.011773 5838 rpcz_store.cc:276] 0507 08:11:58.901792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.904129 (+ 2337us) service_pool.cc:225] Handling call
0507 08:12:00.011514 (+1107385us) ts_tablet_manager.cc:1120] Deleting tablet 072702ef2a6d43248132da6aa3a2d493
0507 08:12:00.011525 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.011606 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1107134}
I20240507 08:12:00.012115 5839 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:00.012156 5838 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 492) took 1110 ms. Trace:
I20240507 08:12:00.012295 5838 rpcz_store.cc:276] 0507 08:11:58.901922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:58.902335 (+ 413us) service_pool.cc:225] Handling call
0507 08:12:00.012016 (+1109681us) ts_tablet_manager.cc:1120] Deleting tablet 831153d2b89e43f8b08b086f7a8f3a54
0507 08:12:00.012026 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:00.012115 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1078170}
I20240507 08:12:00.013208 5838 tablet_replica.cc:330] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.013463 5838 raft_consensus.cc:2229] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.013648 5838 raft_consensus.cc:2258] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.014367 5838 ts_tablet_manager.cc:1875] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.018539 5255 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6a5878e4ea90472ea9b423406f7e0ade
I20240507 08:12:00.018913 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b01d5fc4ce904601b66e4e880c65d768"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.019021 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_35e387a8fca2455ca6def29673fe2449"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.019130 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a9968c4db5a54a7db162505d22dd31dc" } new_table_name: "default.Table_3ca80faa43124b57991879a94bd046cd" modify_external_catalogs: true
W20240507 08:12:00.019510 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35e387a8fca2455ca6def29673fe2449 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.020149 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0a0402c6efc8412aac0d21a545944073
I20240507 08:12:00.020838 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ee5c0609839348ad9fd8706992998487"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.021319 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee5c0609839348ad9fd8706992998487 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:00.022019 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68f1eeccee8e49b3a1dbd1930c2d108a consensus state has no leader
I20240507 08:12:00.023276 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6f66b9533a8d40b093d99097ef61af87" } new_table_name: "default.Table_375f890255ed48659c41d4c11bb30578" modify_external_catalogs: true
I20240507 08:12:00.023991 5838 ts_tablet_manager.cc:1888] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.024128 5838 log.cc:1196] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f922a7519084409db98693ee8797592b
I20240507 08:12:00.024266 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 99e4514039304f0696c69a057db0ea6e
I20240507 08:12:00.027895 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:00.019511 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b01d5fc4ce904601b66e4e880c65d768 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.028399 6410 consensus_queue.cc:1035] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.017s
I20240507 08:12:00.032953 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 198a1826a5344117b7c292823c52ec8e (DEFAULT_TABLE table=default.Table_dac7a1a368f344b9808ba5d8a776b1c9 [id=c85c1a1f38dc4469972e55b3e53467c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.033392 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198a1826a5344117b7c292823c52ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.035262 5843 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.035421 5843 ts_tablet_manager.cc:1367] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.145s user 0.006s sys 0.015s
I20240507 08:12:00.036150 5843 raft_consensus.cc:348] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.036373 5843 raft_consensus.cc:374] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.036525 5843 raft_consensus.cc:729] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.036751 5837 log.cc:1196] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/34bebec088a34ebbb06d2e707d54da4c
I20240507 08:12:00.037202 5837 ts_tablet_manager.cc:1909] T 34bebec088a34ebbb06d2e707d54da4c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.037310 5839 tablet_bootstrap.cc:654] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.038117 5837 tablet_replica.cc:330] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.038328 5837 raft_consensus.cc:2229] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.038465 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc65070bbae4db19c5d14b6670d599c (DEFAULT_TABLE table=default.Table_12604f6607614bb6a9001e40b82cabe3 [id=d3a49a11f5614c43b42cc23cedd8b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.038584 5837 raft_consensus.cc:2258] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.005653 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 1944cb716e7b498a8f13a78b14d8c933 (DEFAULT_TABLE table=default.Table_2e61893054c7404abceb0fc5bc3b08dc [id=7c9835b91edb4cc6b690018d8877459a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.016759 5840 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.030728 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.031075 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 13b3f5952b57428aa1b12b03b104f16d (DEFAULT_TABLE table=default.Table_1a05b4d9237c454db3fb125ff4f6291b [id=5f1a5cc164204bfa9ebe2876d73d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.034083 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 198a1826a5344117b7c292823c52ec8e (DEFAULT_TABLE table=default.Table_dac7a1a368f344b9808ba5d8a776b1c9 [id=c85c1a1f38dc4469972e55b3e53467c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.039530 5840 ts_tablet_manager.cc:1367] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.010s
I20240507 08:12:00.039592 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 34bebec088a34ebbb06d2e707d54da4c (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
I20240507 08:12:00.041515 5840 raft_consensus.cc:348] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.041826 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc65070bbae4db19c5d14b6670d599c (DEFAULT_TABLE table=default.Table_12604f6607614bb6a9001e40b82cabe3 [id=d3a49a11f5614c43b42cc23cedd8b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.041963 5840 raft_consensus.cc:374] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.042114 5840 raft_consensus.cc:729] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.042219 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc65070bbae4db19c5d14b6670d599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.042603 5840 consensus_queue.cc:260] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.046649 5841 ts_tablet_manager.cc:1888] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.048099 5841 log.cc:1196] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/fde460fdcf254af6a3cdaaac22ac2d93
I20240507 08:12:00.048621 5841 ts_tablet_manager.cc:1909] T fde460fdcf254af6a3cdaaac22ac2d93 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.049015 5840 ts_tablet_manager.cc:1398] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240507 08:12:00.049604 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 1944cb716e7b498a8f13a78b14d8c933 (DEFAULT_TABLE table=default.Table_2e61893054c7404abceb0fc5bc3b08dc [id=7c9835b91edb4cc6b690018d8877459a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.050529 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet fde460fdcf254af6a3cdaaac22ac2d93 (table default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]) successfully deleted
I20240507 08:12:00.052783 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944cb716e7b498a8f13a78b14d8c933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.053334 5840 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.034366 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0402c6efc8412aac0d21a545944073 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.038856 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc65070bbae4db19c5d14b6670d599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.054688 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:00.057257 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ab94b1a09dd4969b928b017229ce4bf consensus state has no leader
I20240507 08:12:00.058130 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198a1826a5344117b7c292823c52ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.061787 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b3f5952b57428aa1b12b03b104f16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.052372 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 3a065a4441f449a299515dda078f2d24 (DEFAULT_TABLE table=default.Table_0ae24aad0d2044a98f1f464c8a31d715 [id=8db1f44776ee479aadd2ab90e091de1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.049403 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 13b3f5952b57428aa1b12b03b104f16d (DEFAULT_TABLE table=default.Table_1a05b4d9237c454db3fb125ff4f6291b [id=5f1a5cc164204bfa9ebe2876d73d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.049810 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0402c6efc8412aac0d21a545944073 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.070926 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a065a4441f449a299515dda078f2d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.073946 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944cb716e7b498a8f13a78b14d8c933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.077855 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b3f5952b57428aa1b12b03b104f16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.078325 5840 tablet_bootstrap.cc:654] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.079514 5843 consensus_queue.cc:260] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:00.081497 6379 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20240507 08:12:00.087622 5837 ts_tablet_manager.cc:1875] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:00.090231 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6b4c69d38e484f5c819b7dd915bf4410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4c69d38e484f5c819b7dd915bf4410 already in progress: creating tablet
I20240507 08:12:00.088887 5840 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.090564 5840 ts_tablet_manager.cc:1367] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
W20240507 08:12:00.091621 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 03a59c7257e74b4abaa4ea7a1ee9136d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03a59c7257e74b4abaa4ea7a1ee9136d already in progress: creating tablet
I20240507 08:12:00.091802 5838 ts_tablet_manager.cc:1909] T f922a7519084409db98693ee8797592b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.093276 5838 tablet_replica.cc:330] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.093472 5838 raft_consensus.cc:2229] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.098595 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 87377661d9374cef8ab1a97042b7f1b9 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.098841 5840 raft_consensus.cc:348] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.099074 5840 raft_consensus.cc:374] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.099189 5840 raft_consensus.cc:729] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.101233 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:00.105754 5838 raft_consensus.cc:2258] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.106551 5838 ts_tablet_manager.cc:1875] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:00.106698 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d1c8db73b8cb4212b59d40bd6864e23f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1c8db73b8cb4212b59d40bd6864e23f already in progress: creating tablet
I20240507 08:12:00.107584 5843 ts_tablet_manager.cc:1398] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240507 08:12:00.108316 5843 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.112555 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0ae24aad0d2044a98f1f464c8a31d715" } modify_external_catalogs: true
I20240507 08:12:00.112716 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e61893054c7404abceb0fc5bc3b08dc" } new_table_name: "default.Table_6549c56b83e24fe49342c6864c56162c" modify_external_catalogs: true
I20240507 08:12:00.112841 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0ae24aad0d2044a98f1f464c8a31d715" } modify_external_catalogs: true
I20240507 08:12:00.099483 5840 consensus_queue.cc:260] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.115165 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f4a5ad85280b49289eeef7eea3c2849d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.115820 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f4a5ad85280b49289eeef7eea3c2849d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:00.117373 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 99e4514039304f0696c69a057db0ea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99e4514039304f0696c69a057db0ea6e already in progress: creating tablet
I20240507 08:12:00.118937 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:00.122020 5840 ts_tablet_manager.cc:1398] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240507 08:12:00.122342 5840 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.123958 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.135401 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.135831 5841 tablet_replica.cc:330] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.135836 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.136808 5841 raft_consensus.cc:2229] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:00.138492 5837 ts_tablet_manager.cc:1888] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:00.139334 5840 tablet_bootstrap.cc:654] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.146338 5837 log.cc:1196] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1f5d566695d94c708e90ee30254084a1
I20240507 08:12:00.146719 5841 raft_consensus.cc:2258] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.146796 5837 ts_tablet_manager.cc:1909] T 1f5d566695d94c708e90ee30254084a1 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.148016 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1f5d566695d94c708e90ee30254084a1 (table default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]) successfully deleted
W20240507 08:12:00.148833 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1f43a80d38ed4788a13421b245fe6e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f43a80d38ed4788a13421b245fe6e8b already in progress: creating tablet
W20240507 08:12:00.149479 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f513573e340547d6ae5ab848d00ace72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f513573e340547d6ae5ab848d00ace72 already in progress: creating tablet
I20240507 08:12:00.149848 5837 tablet_replica.cc:330] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.150261 5837 raft_consensus.cc:2229] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.150691 5837 raft_consensus.cc:2258] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.154377 5841 ts_tablet_manager.cc:1875] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.157433 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:00.160292 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 972bc0a5ba924e0fb946cdacd9575c1e (table default.Table_cebf0395fd3b4cf1a49263069e98ae91 [id=298d01ccedc24341a84f7c06d5241b11]) successfully deleted
W20240507 08:12:00.160701 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c84a2018f1c442e2a4dfc1c6b9c5ceef already in progress: creating tablet
W20240507 08:12:00.161206 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 63ccdad494794192b3c37c8d5cbfcb17 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63ccdad494794192b3c37c8d5cbfcb17 already in progress: creating tablet
W20240507 08:12:00.161793 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 53fb38b567c5455ebc441e5f71db694f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53fb38b567c5455ebc441e5f71db694f already in progress: creating tablet
W20240507 08:12:00.162211 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5e0fb75933874a948281bf573a1723a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5e0fb75933874a948281bf573a1723a1 already in progress: creating tablet
W20240507 08:12:00.162586 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 167ee9cbc9174b82824c468e098472a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 167ee9cbc9174b82824c468e098472a6 already in progress: creating tablet
W20240507 08:12:00.162989 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4e5439844f384513b4719cfe4f4e5d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e5439844f384513b4719cfe4f4e5d9b already in progress: creating tablet
W20240507 08:12:00.163359 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0d617213b959452ab6519809e56db128 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d617213b959452ab6519809e56db128 already in progress: creating tablet
W20240507 08:12:00.163774 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 072702ef2a6d43248132da6aa3a2d493 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 072702ef2a6d43248132da6aa3a2d493 already in progress: creating tablet
W20240507 08:12:00.164274 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 831153d2b89e43f8b08b086f7a8f3a54 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 831153d2b89e43f8b08b086f7a8f3a54 already in progress: creating tablet
W20240507 08:12:00.164693 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 190baa48859640cf928642dccbebfceb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 190baa48859640cf928642dccbebfceb already in progress: creating tablet
W20240507 08:12:00.165055 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aca8c38841db45e38ae5d8d965a16d0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aca8c38841db45e38ae5d8d965a16d0e already in progress: creating tablet
W20240507 08:12:00.165509 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a3e58ba8c26d4f17af24eb379aca1f60 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3e58ba8c26d4f17af24eb379aca1f60 already in progress: creating tablet
I20240507 08:12:00.165896 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f922a7519084409db98693ee8797592b (table default.Table_0aeed0fe490a4bd4b753a3c37a0cad6e [id=6d259677b27c47aea4c5e5cb54f1a02f]) successfully deleted
W20240507 08:12:00.166291 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 916e33e40ef44d06a359b82d8fe48157 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 916e33e40ef44d06a359b82d8fe48157 already in progress: creating tablet
W20240507 08:12:00.166709 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6b4c69d38e484f5c819b7dd915bf4410 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6b4c69d38e484f5c819b7dd915bf4410 already in progress: creating tablet
W20240507 08:12:00.167165 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 156bf04b57724134a45fa9fc477596c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 156bf04b57724134a45fa9fc477596c5 already in progress: creating tablet
W20240507 08:12:00.167515 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b31fdadee60e4f5e94e92c536e18774d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b31fdadee60e4f5e94e92c536e18774d already in progress: creating tablet
W20240507 08:12:00.168012 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f513573e340547d6ae5ab848d00ace72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f513573e340547d6ae5ab848d00ace72 already in progress: creating tablet
W20240507 08:12:00.168707 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1f43a80d38ed4788a13421b245fe6e8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f43a80d38ed4788a13421b245fe6e8b already in progress: creating tablet
W20240507 08:12:00.169016 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a5976eca006448e0b9f2573acd738bb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5976eca006448e0b9f2573acd738bb1 already in progress: creating tablet
I20240507 08:12:00.171262 5843 tablet_bootstrap.cc:654] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.178105 5837 ts_tablet_manager.cc:1875] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.178105 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 3a065a4441f449a299515dda078f2d24 (DEFAULT_TABLE table=default.Table_0ae24aad0d2044a98f1f464c8a31d715 [id=8db1f44776ee479aadd2ab90e091de1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.178927 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a065a4441f449a299515dda078f2d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.183485 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c8db73b8cb4212b59d40bd6864e23f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.186620 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 2013559307a34443ac998dab956c55f9 (DEFAULT_TABLE table=default.Table_f38a29b591904253b1165f8512dce2a7 [id=c602a858547f42c583e14e99a99c4a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.189785 6410 consensus_queue.cc:1035] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:00.191069 5838 ts_tablet_manager.cc:1888] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.191716 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013559307a34443ac998dab956c55f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.196698 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 2013559307a34443ac998dab956c55f9 (DEFAULT_TABLE table=default.Table_f38a29b591904253b1165f8512dce2a7 [id=c602a858547f42c583e14e99a99c4a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.197256 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a778d8595fc7496bace57c63194455fe" } modify_external_catalogs: true
I20240507 08:12:00.197458 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a778d8595fc7496bace57c63194455fe" } modify_external_catalogs: true
I20240507 08:12:00.197706 5841 ts_tablet_manager.cc:1888] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.197837 5841 log.cc:1196] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/0a0402c6efc8412aac0d21a545944073
I20240507 08:12:00.198256 5841 ts_tablet_manager.cc:1909] T 0a0402c6efc8412aac0d21a545944073 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.198902 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3b2b1ae34b854b27b9a0ab00c78c17bb" } new_table_name: "default.Table_7007a63c36ea47c78769b5bcafec4cf9" modify_external_catalogs: true
I20240507 08:12:00.199335 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_e85ccc16434742ac812466c315098b9a [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]) successfully deleted
I20240507 08:12:00.200389 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 7a613de37e954961a9356e28db5e629c (DEFAULT_TABLE table=default.Table_33512c5a59404199a0ac2d5ff4e78bba [id=a79ba558b19f43128f991a904c100f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.200352 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e292ef7310c5489f8b3c60c61ab75b92"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.200801 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a613de37e954961a9356e28db5e629c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:00.200873 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e292ef7310c5489f8b3c60c61ab75b92 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.203406 5843 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.203569 5843 ts_tablet_manager.cc:1367] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.002s
W20240507 08:12:00.204274 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 323b96ae7d1e471998a39f3be0857ce7 has no consensus state
I20240507 08:12:00.204321 5843 raft_consensus.cc:348] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.204594 5843 raft_consensus.cc:374] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.204677 5843 raft_consensus.cc:729] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.204949 5843 consensus_queue.cc:260] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.205474 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013559307a34443ac998dab956c55f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.211633 5843 ts_tablet_manager.cc:1398] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:12:00.212671 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 7a613de37e954961a9356e28db5e629c (DEFAULT_TABLE table=default.Table_33512c5a59404199a0ac2d5ff4e78bba [id=a79ba558b19f43128f991a904c100f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.213070 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a613de37e954961a9356e28db5e629c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.215353 6426 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:00.219815 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.221499 5417 tablet_service.cc:1467] Processing CreateTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 (DEFAULT_TABLE table=default.Table_81faed45055c4c6c9d1cb2efbd2993a2 [id=3853bef75cfe4a468f8060ac74a930f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.221953 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3e1823f9ff4f19be2b13cdcd706f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.225682 5840 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.225852 5840 ts_tablet_manager.cc:1367] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.104s user 0.008s sys 0.009s
I20240507 08:12:00.226603 5840 raft_consensus.cc:348] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.226827 5840 raft_consensus.cc:374] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.226905 5840 raft_consensus.cc:729] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.227169 5840 consensus_queue.cc:260] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.227840 5840 ts_tablet_manager.cc:1398] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:00.231693 5840 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.233785 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3a065a4441f449a299515dda078f2d24
I20240507 08:12:00.234580 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a065a4441f449a299515dda078f2d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.235961 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4d08d2f517e6420aa76bcf5ec7928959"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.236485 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d08d2f517e6420aa76bcf5ec7928959 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:00.238644 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1944cb716e7b498a8f13a78b14d8c933 consensus state has no leader
I20240507 08:12:00.238584 5698 tablet_service.cc:1467] Processing CreateTablet for tablet e07927ac3ef9492292335d7783fdc758 (DEFAULT_TABLE table=default.Table_bd945626c197488da2bb9d0d8e8a55dd [id=de2375b8c3534be2ac331a08fcd16643]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.238997 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07927ac3ef9492292335d7783fdc758. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.238997 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0357417146364abdb06bb955b43a5fe2" } modify_external_catalogs: true
I20240507 08:12:00.239148 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0357417146364abdb06bb955b43a5fe2" } modify_external_catalogs: true
I20240507 08:12:00.240267 5843 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.240645 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de" } modify_external_catalogs: true
I20240507 08:12:00.240814 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de" } modify_external_catalogs: true
I20240507 08:12:00.241483 6396 catalog_manager.cc:1259] Loaded cluster ID: 5ffc45d58e0243d3a7f54e418cabbd00
I20240507 08:12:00.241600 6396 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: loading cluster ID for follower catalog manager: success
I20240507 08:12:00.241839 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0bba3367e45542dcb9de73f0f9a05a6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.242306 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bba3367e45542dcb9de73f0f9a05a6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.242905 5841 tablet_replica.cc:330] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.244271 5415 tablet_service.cc:1467] Processing CreateTablet for tablet e07927ac3ef9492292335d7783fdc758 (DEFAULT_TABLE table=default.Table_bd945626c197488da2bb9d0d8e8a55dd [id=de2375b8c3534be2ac331a08fcd16643]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.244400 5841 raft_consensus.cc:2229] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.244446 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a3e97a9c990549b880b71e973f807616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.244623 5841 raft_consensus.cc:2258] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.244876 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07927ac3ef9492292335d7783fdc758. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:00.245129 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3e97a9c990549b880b71e973f807616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.246074 5691 tablet_service.cc:1467] Processing CreateTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 (DEFAULT_TABLE table=default.Table_81faed45055c4c6c9d1cb2efbd2993a2 [id=3853bef75cfe4a468f8060ac74a930f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.246459 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3e1823f9ff4f19be2b13cdcd706f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.249629 5843 tablet_bootstrap.cc:654] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.250366 5841 ts_tablet_manager.cc:1875] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.251286 6432 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:00.253803 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a065a4441f449a299515dda078f2d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.193533 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.195134 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 4867e1cca1634b4ab6e36f9ed3204e0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.195698 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 1944cb716e7b498a8f13a78b14d8c933 (DEFAULT_TABLE table=default.Table_2e61893054c7404abceb0fc5bc3b08dc [id=7c9835b91edb4cc6b690018d8877459a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.196041 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 13b3f5952b57428aa1b12b03b104f16d (DEFAULT_TABLE table=default.Table_1a05b4d9237c454db3fb125ff4f6291b [id=5f1a5cc164204bfa9ebe2876d73d865e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.197665 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0402c6efc8412aac0d21a545944073 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.199193 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.210115 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 198a1826a5344117b7c292823c52ec8e (DEFAULT_TABLE table=default.Table_dac7a1a368f344b9808ba5d8a776b1c9 [id=c85c1a1f38dc4469972e55b3e53467c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.212867 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 8fc65070bbae4db19c5d14b6670d599c (DEFAULT_TABLE table=default.Table_12604f6607614bb6a9001e40b82cabe3 [id=d3a49a11f5614c43b42cc23cedd8b483]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.264518 5840 tablet_bootstrap.cc:654] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.270561 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13b3f5952b57428aa1b12b03b104f16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.272635 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1944cb716e7b498a8f13a78b14d8c933. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.274714 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fc65070bbae4db19c5d14b6670d599c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.275504 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 198a1826a5344117b7c292823c52ec8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.278774 5837 ts_tablet_manager.cc:1888] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.282696 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 3a065a4441f449a299515dda078f2d24 (DEFAULT_TABLE table=default.Table_0ae24aad0d2044a98f1f464c8a31d715 [id=8db1f44776ee479aadd2ab90e091de1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.282928 5837 log.cc:1196] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/fde460fdcf254af6a3cdaaac22ac2d93
I20240507 08:12:00.283360 5837 ts_tablet_manager.cc:1909] T fde460fdcf254af6a3cdaaac22ac2d93 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.283145 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2013559307a34443ac998dab956c55f9 (DEFAULT_TABLE table=default.Table_f38a29b591904253b1165f8512dce2a7 [id=c602a858547f42c583e14e99a99c4a39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.274590 5838 log.cc:1196] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/270a1fd15925416483c211021549315c
I20240507 08:12:00.284207 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a065a4441f449a299515dda078f2d24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.285471 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a065a4441f449a299515dda078f2d24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.285841 5553 tablet_service.cc:1467] Processing CreateTablet for tablet e07927ac3ef9492292335d7783fdc758 (DEFAULT_TABLE table=default.Table_bd945626c197488da2bb9d0d8e8a55dd [id=de2375b8c3534be2ac331a08fcd16643]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.286224 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e07927ac3ef9492292335d7783fdc758. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:00.287348 5517 leader_election.cc:341] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 4867e1cca1634b4ab6e36f9ed3204e0a
I20240507 08:12:00.287618 5517 leader_election.cc:302] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:00.289901 5378 consensus_peers.cc:485] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:00.295646 5841 ts_tablet_manager.cc:1888] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.295806 5841 log.cc:1196] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/072702ef2a6d43248132da6aa3a2d493
I20240507 08:12:00.296173 5841 ts_tablet_manager.cc:1909] T 072702ef2a6d43248132da6aa3a2d493 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.297214 5841 tablet_replica.cc:330] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.297416 5841 raft_consensus.cc:2229] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.297608 5841 raft_consensus.cc:2258] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.298269 5841 ts_tablet_manager.cc:1875] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.299288 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 072702ef2a6d43248132da6aa3a2d493 (table default.Table_c6bf901b2c4a491e8aa4c357670a5d08 [id=3d99e1ae7dc34ac49843ce5f159f891e]) successfully deleted
I20240507 08:12:00.300410 6396 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: acquiring CA information for follower catalog manager: success
I20240507 08:12:00.301462 5837 tablet_replica.cc:330] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.301821 5837 raft_consensus.cc:2229] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.301993 5837 raft_consensus.cc:2258] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.305645 5837 ts_tablet_manager.cc:1875] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.289192 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2013559307a34443ac998dab956c55f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.295238 5556 tablet_service.cc:1467] Processing CreateTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 (DEFAULT_TABLE table=default.Table_81faed45055c4c6c9d1cb2efbd2993a2 [id=3853bef75cfe4a468f8060ac74a930f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.295434 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 7a613de37e954961a9356e28db5e629c (DEFAULT_TABLE table=default.Table_33512c5a59404199a0ac2d5ff4e78bba [id=a79ba558b19f43128f991a904c100f42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.314956 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet fde460fdcf254af6a3cdaaac22ac2d93 (table default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]) successfully deleted
I20240507 08:12:00.315085 5837 ts_tablet_manager.cc:1888] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.315234 5837 log.cc:1196] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/88a2cfb5f046434c8340234a447a7e65
I20240507 08:12:00.315672 5837 ts_tablet_manager.cc:1909] T 88a2cfb5f046434c8340234a447a7e65 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.316967 6418 raft_consensus.cc:2726] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:00.318113 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 88a2cfb5f046434c8340234a447a7e65 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
W20240507 08:12:00.318490 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 03a59c7257e74b4abaa4ea7a1ee9136d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03a59c7257e74b4abaa4ea7a1ee9136d already in progress: creating tablet
I20240507 08:12:00.318645 5840 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.318802 5840 ts_tablet_manager.cc:1367] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.087s user 0.001s sys 0.003s
W20240507 08:12:00.318954 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d1c8db73b8cb4212b59d40bd6864e23f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1c8db73b8cb4212b59d40bd6864e23f already in progress: creating tablet
I20240507 08:12:00.319608 5840 raft_consensus.cc:348] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.319815 5840 raft_consensus.cc:374] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.319921 5840 raft_consensus.cc:729] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.320220 5840 consensus_queue.cc:260] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.323078 5840 ts_tablet_manager.cc:1398] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:12:00.323370 5840 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.325732 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a613de37e954961a9356e28db5e629c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.330772 5840 tablet_bootstrap.cc:654] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:00.333917 5378 consensus_peers.cc:485] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:00.333954 5837 tablet_replica.cc:330] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.334213 5837 raft_consensus.cc:2229] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.334432 5837 raft_consensus.cc:2258] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.335706 5837 ts_tablet_manager.cc:1875] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.338433 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec3e1823f9ff4f19be2b13cdcd706f59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.338927 5838 ts_tablet_manager.cc:1909] T 270a1fd15925416483c211021549315c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:00.339314 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 99e4514039304f0696c69a057db0ea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99e4514039304f0696c69a057db0ea6e already in progress: creating tablet
I20240507 08:12:00.344077 6376 raft_consensus.cc:482] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.344168 5837 ts_tablet_manager.cc:1888] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.344317 5837 log.cc:1196] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:12:00.344249 6376 raft_consensus.cc:504] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.344729 5837 ts_tablet_manager.cc:1909] T 7eaf0e2936324a9fbb2426a8026886fc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.344974 6376 leader_election.cc:288] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:00.345623 5837 tablet_replica.cc:330] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.345904 5837 raft_consensus.cc:2229] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.346141 5837 raft_consensus.cc:2258] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.346129 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554ebc621c24d459668f2cefc20b641" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:00.346638 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5554ebc621c24d459668f2cefc20b641" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:00.346829 5656 leader_election.cc:341] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.347165 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 7eaf0e2936324a9fbb2426a8026886fc (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:12:00.347342 5837 ts_tablet_manager.cc:1875] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:00.347380 5654 leader_election.cc:341] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.347524 5654 leader_election.cc:302] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.347755 6376 raft_consensus.cc:2726] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:00.352843 5840 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.353022 5840 ts_tablet_manager.cc:1367] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20240507 08:12:00.353853 5840 raft_consensus.cc:348] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.354107 5840 raft_consensus.cc:374] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.354187 5840 raft_consensus.cc:729] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.354429 5837 ts_tablet_manager.cc:1888] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.354629 5837 log.cc:1196] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/0a0402c6efc8412aac0d21a545944073
I20240507 08:12:00.355142 5837 ts_tablet_manager.cc:1909] T 0a0402c6efc8412aac0d21a545944073 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.354512 5840 consensus_queue.cc:260] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.356017 5840 ts_tablet_manager.cc:1398] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:00.356380 5840 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.358127 5840 tablet_bootstrap.cc:654] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.361783 5837 tablet_replica.cc:330] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.362038 5837 raft_consensus.cc:2229] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.362243 5837 raft_consensus.cc:2258] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.363168 5837 ts_tablet_manager.cc:1875] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.365790 5838 tablet_replica.cc:330] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.366048 5838 raft_consensus.cc:2229] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.366325 5838 raft_consensus.cc:2258] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.367056 5838 ts_tablet_manager.cc:1875] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.370690 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_e85ccc16434742ac812466c315098b9a [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]) successfully deleted
I20240507 08:12:00.374569 5837 ts_tablet_manager.cc:1888] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.374866 5837 log.cc:1196] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5e53409fe13e4a9585dbba648a25cc07
I20240507 08:12:00.375427 5837 ts_tablet_manager.cc:1909] T 5e53409fe13e4a9585dbba648a25cc07 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.377400 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5e53409fe13e4a9585dbba648a25cc07 (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:12:00.377792 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 270a1fd15925416483c211021549315c (table default.Table_169e8150dda24f87ac304a6c9031909d [id=4189d27df0684371941ce5bc86a70c46]) successfully deleted
W20240507 08:12:00.378386 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3a065a4441f449a299515dda078f2d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a065a4441f449a299515dda078f2d24 already in progress: creating tablet
W20240507 08:12:00.378481 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1f5d566695d94c708e90ee30254084a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f5d566695d94c708e90ee30254084a1 already in progress: creating tablet
W20240507 08:12:00.378846 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d1c8db73b8cb4212b59d40bd6864e23f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1c8db73b8cb4212b59d40bd6864e23f already in progress: creating tablet
W20240507 08:12:00.379388 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 03a59c7257e74b4abaa4ea7a1ee9136d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03a59c7257e74b4abaa4ea7a1ee9136d already in progress: creating tablet
I20240507 08:12:00.381139 5839 tablet_bootstrap.cc:492] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:00.384433 5841 ts_tablet_manager.cc:1888] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.384564 5841 log.cc:1196] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:12:00.384876 5841 ts_tablet_manager.cc:1909] T f5e48a2b4a0e4088ad009a7f8fe77f3f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.386404 5843 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.386538 5843 ts_tablet_manager.cc:1367] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.147s user 0.001s sys 0.002s
I20240507 08:12:00.387413 5843 raft_consensus.cc:348] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.388134 5843 raft_consensus.cc:374] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.388211 5843 raft_consensus.cc:729] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.388417 5843 consensus_queue.cc:260] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.386399 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 06898414c5494cd380e0c932f4589014 (DEFAULT_TABLE table=default.Table_761e0fb1290f435f82ab49c01e433ad3 [id=b42c617c6a1f449694b15fd83c908d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.388909 5843 ts_tablet_manager.cc:1398] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:00.389228 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
W20240507 08:12:00.389621 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3a065a4441f449a299515dda078f2d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a065a4441f449a299515dda078f2d24 already in progress: creating tablet
I20240507 08:12:00.389827 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 06898414c5494cd380e0c932f4589014 (DEFAULT_TABLE table=default.Table_761e0fb1290f435f82ab49c01e433ad3 [id=b42c617c6a1f449694b15fd83c908d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.389993 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06898414c5494cd380e0c932f4589014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.390550 5843 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.392940 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06898414c5494cd380e0c932f4589014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.387768 6441 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:00.398977 6396 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20240507 08:12:00.400882 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 8c383c49d5974eee9f19cedb43fa812f (DEFAULT_TABLE table=default.Table_a894109d6af4424db5114275d258a4f0 [id=f148283c2b5145f68a59698098641bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.401185 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 8c383c49d5974eee9f19cedb43fa812f (DEFAULT_TABLE table=default.Table_a894109d6af4424db5114275d258a4f0 [id=f148283c2b5145f68a59698098641bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.401612 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c383c49d5974eee9f19cedb43fa812f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.409245 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 06898414c5494cd380e0c932f4589014 (DEFAULT_TABLE table=default.Table_761e0fb1290f435f82ab49c01e433ad3 [id=b42c617c6a1f449694b15fd83c908d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.409751 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06898414c5494cd380e0c932f4589014. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.418623 5843 tablet_bootstrap.cc:654] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.418697 5839 ts_tablet_manager.cc:1367] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.407s user 0.004s sys 0.004s
I20240507 08:12:00.419799 5839 raft_consensus.cc:348] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.420096 5839 raft_consensus.cc:374] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.420214 5839 raft_consensus.cc:729] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.420490 5839 consensus_queue.cc:260] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.428339 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3a3f56ea5b7a43e4a89845da5a4d98f7
I20240507 08:12:00.429024 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 deleted and replaced by tablet c9a65a8e939f4b7d89d00720845ea1fa
I20240507 08:12:00.429962 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c9a65a8e939f4b7d89d00720845ea1fa) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.430256 5841 tablet_replica.cc:330] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.430470 5841 raft_consensus.cc:2229] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:00.430790 5841 raft_consensus.cc:2258] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.431375 5841 ts_tablet_manager.cc:1875] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.436386 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:00.437189 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bf6fa5a914c14039a8f8940d2197f70a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.437896 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf6fa5a914c14039a8f8940d2197f70a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.438551 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c9a65a8e939f4b7d89d00720845ea1fa) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.438925 5837 tablet_replica.cc:330] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.439170 5837 raft_consensus.cc:2229] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.439414 5837 raft_consensus.cc:2258] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.439993 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3c7f072f55a94cca938aa063637af9a5
I20240507 08:12:00.440066 5837 ts_tablet_manager.cc:1875] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.440805 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dfc198b816c048acb4142d4ede67abb7" } new_table_name: "default.Table_8493d14dd0f8480591e6e0a0196c77cb" modify_external_catalogs: true
I20240507 08:12:00.441210 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7f072f55a94cca938aa063637af9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.441512 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7f072f55a94cca938aa063637af9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.442281 5841 ts_tablet_manager.cc:1888] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.445009 5841 log.cc:1196] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3a3f56ea5b7a43e4a89845da5a4d98f7
I20240507 08:12:00.445444 5841 ts_tablet_manager.cc:1909] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.446420 5837 ts_tablet_manager.cc:1888] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.446610 5837 log.cc:1196] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/3a3f56ea5b7a43e4a89845da5a4d98f7
I20240507 08:12:00.447052 5837 ts_tablet_manager.cc:1909] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.448123 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]) successfully deleted
W20240507 08:12:00.448634 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3c7f072f55a94cca938aa063637af9a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c7f072f55a94cca938aa063637af9a5 already in progress: creating tablet
I20240507 08:12:00.452602 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]) successfully deleted
I20240507 08:12:00.454015 6376 raft_consensus.cc:482] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.454159 6376 raft_consensus.cc:504] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.454890 6376 leader_election.cc:288] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:00.455845 5656 leader_election.cc:341] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 6bf34089e5ec419f97c0da9ec42d7eec
W20240507 08:12:00.458099 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3c7f072f55a94cca938aa063637af9a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c7f072f55a94cca938aa063637af9a5 already in progress: creating tablet
I20240507 08:12:00.459017 5843 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.459434 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 8c383c49d5974eee9f19cedb43fa812f (DEFAULT_TABLE table=default.Table_a894109d6af4424db5114275d258a4f0 [id=f148283c2b5145f68a59698098641bf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.459913 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c383c49d5974eee9f19cedb43fa812f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.460888 5843 ts_tablet_manager.cc:1367] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.005s
I20240507 08:12:00.462297 5408 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd191b107544d88e62f31c79e254b4 (DEFAULT_TABLE table=default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de [id=8d27529a01c2475c862d26f344ec2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.462697 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd191b107544d88e62f31c79e254b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.462638 5687 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd191b107544d88e62f31c79e254b4 (DEFAULT_TABLE table=default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de [id=8d27529a01c2475c862d26f344ec2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.463022 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd191b107544d88e62f31c79e254b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.463639 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 (DEFAULT_TABLE table=default.Table_a07279891494410096ed3d7da872b203 [id=76f5f0241cad4c09af24b15b3866bdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.464042 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3ea4658bb946439e4a50962444b3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.469605 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 (DEFAULT_TABLE table=default.Table_a07279891494410096ed3d7da872b203 [id=76f5f0241cad4c09af24b15b3866bdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.469947 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3ea4658bb946439e4a50962444b3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.461663 5843 raft_consensus.cc:348] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.473927 5843 raft_consensus.cc:374] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.474066 5843 raft_consensus.cc:729] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.474750 5843 consensus_queue.cc:260] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:00.474989 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35e34b60b78b402faf51e003c97db787 consensus state has no leader
I20240507 08:12:00.476774 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7" } modify_external_catalogs: true
I20240507 08:12:00.477000 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7" } modify_external_catalogs: true
I20240507 08:12:00.478839 6409 raft_consensus.cc:482] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.478973 6409 raft_consensus.cc:504] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.479784 6409 leader_election.cc:288] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:00.480839 5377 leader_election.cc:341] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 34bebec088a34ebbb06d2e707d54da4c
I20240507 08:12:00.483266 5843 ts_tablet_manager.cc:1398] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240507 08:12:00.486032 5843 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.487758 5843 tablet_bootstrap.cc:654] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.499493 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:00.499863 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:00.506994 5840 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.507146 5840 ts_tablet_manager.cc:1367] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.151s user 0.046s sys 0.048s
I20240507 08:12:00.507941 5840 raft_consensus.cc:348] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.508352 5840 raft_consensus.cc:374] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.508456 5840 raft_consensus.cc:729] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.508702 5840 consensus_queue.cc:260] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.509356 5840 ts_tablet_manager.cc:1398] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:00.509768 5840 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.432328 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 with delete_type TABLET_DATA_DELETED (replaced by tablet c9a65a8e939f4b7d89d00720845ea1fa) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.444140 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7f072f55a94cca938aa063637af9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.453395 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c383c49d5974eee9f19cedb43fa812f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.511395 5840 tablet_bootstrap.cc:654] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.456820 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6bf34089e5ec419f97c0da9ec42d7eec" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.481586 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34bebec088a34ebbb06d2e707d54da4c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.487507 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet f49dd4772f8e46478638b29c009c83ea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:00.513540 5654 leader_election.cc:341] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.513728 5654 leader_election.cc:302] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.514159 6376 raft_consensus.cc:2726] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:00.519543 5840 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.519706 5840 ts_tablet_manager.cc:1367] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.003s
I20240507 08:12:00.520395 5840 raft_consensus.cc:348] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.520591 5840 raft_consensus.cc:374] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.520661 5840 raft_consensus.cc:729] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.520900 5840 consensus_queue.cc:260] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.522581 5840 ts_tablet_manager.cc:1398] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:00.522871 5840 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.524351 5840 tablet_bootstrap.cc:654] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.524891 5838 ts_tablet_manager.cc:1888] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.525087 5838 log.cc:1196] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4867e1cca1634b4ab6e36f9ed3204e0a
W20240507 08:12:00.530577 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 35e34b60b78b402faf51e003c97db787 (table default.Table_7007a63c36ea47c78769b5bcafec4cf9 [id=71fd821e13734824a24f76fa2c220f1a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 35e34b60b78b402faf51e003c97db787 consensus state has no leader
W20240507 08:12:00.531863 5378 leader_election.cc:341] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.532024 5378 leader_election.cc:302] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.532406 5995 raft_consensus.cc:2726] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:00.532469 6409 raft_consensus.cc:482] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.532616 6409 raft_consensus.cc:504] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.533227 6409 leader_election.cc:288] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.533998 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:00.534202 5718 raft_consensus.cc:2454] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:00.534644 5377 leader_election.cc:302] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:00.535013 6409 raft_consensus.cc:2781] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:00.535096 6409 raft_consensus.cc:482] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:00.535184 6409 raft_consensus.cc:3037] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.537861 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 1d432a59bdd344ddaaf710de73b543e6 (DEFAULT_TABLE table=default.Table_22d1db9ba344416091aab96381212e34 [id=cb560b34838b48bfb987e60ea243f9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.538246 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d432a59bdd344ddaaf710de73b543e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.538331 6376 raft_consensus.cc:482] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.538492 6376 raft_consensus.cc:504] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.539117 6376 leader_election.cc:288] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:00.539212 5417 tablet_service.cc:1467] Processing CreateTablet for tablet e3078f2bf97d42d4bde80035e04a47a6 (DEFAULT_TABLE table=default.Table_ad2e433251d046b589865483014c790e [id=88e838684d214496826cb851041ee2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.539566 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3078f2bf97d42d4bde80035e04a47a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.539742 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:00.540338 5656 leader_election.cc:341] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.541853 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 1d432a59bdd344ddaaf710de73b543e6 (DEFAULT_TABLE table=default.Table_22d1db9ba344416091aab96381212e34 [id=cb560b34838b48bfb987e60ea243f9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.542397 6409 raft_consensus.cc:504] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.542762 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d432a59bdd344ddaaf710de73b543e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.543704 6409 leader_election.cc:288] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.544178 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:00.544404 5718 raft_consensus.cc:3037] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.547096 5718 raft_consensus.cc:2454] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:00.547520 5377 leader_election.cc:302] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:00.547822 6409 raft_consensus.cc:2781] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:00.548009 6409 raft_consensus.cc:686] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:00.548282 6409 consensus_queue.cc:237] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.555181 5843 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.555336 5843 ts_tablet_manager.cc:1367] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.001s
I20240507 08:12:00.556077 5843 raft_consensus.cc:348] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.556378 5843 raft_consensus.cc:374] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.556471 5843 raft_consensus.cc:729] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.558192 5843 consensus_queue.cc:260] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.558733 5843 ts_tablet_manager.cc:1398] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:00.559013 5843 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.566951 5691 tablet_service.cc:1467] Processing CreateTablet for tablet e3078f2bf97d42d4bde80035e04a47a6 (DEFAULT_TABLE table=default.Table_ad2e433251d046b589865483014c790e [id=88e838684d214496826cb851041ee2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.567384 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3078f2bf97d42d4bde80035e04a47a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.569612 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a
I20240507 08:12:00.571027 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.571511 5843 tablet_bootstrap.cc:654] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.572489 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:00.572916 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a already in progress: creating tablet
W20240507 08:12:00.574635 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a already in progress: creating tablet
I20240507 08:12:00.575660 5412 tablet_service.cc:1467] Processing CreateTablet for tablet e7417948f54a490892e083caeddad6af (DEFAULT_TABLE table=default.Table_659bd38568664e459c30bf2ad102ff78 [id=50e6da1e1865462da3f718a2793c6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.576036 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7417948f54a490892e083caeddad6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.577966 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ff9a2c687f9446dab1f59c2df5d949bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.578476 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff9a2c687f9446dab1f59c2df5d949bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.580555 5687 tablet_service.cc:1467] Processing CreateTablet for tablet e7417948f54a490892e083caeddad6af (DEFAULT_TABLE table=default.Table_659bd38568664e459c30bf2ad102ff78 [id=50e6da1e1865462da3f718a2793c6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.580917 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7417948f54a490892e083caeddad6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.582089 5838 ts_tablet_manager.cc:1909] T 4867e1cca1634b4ab6e36f9ed3204e0a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.421562 5839 ts_tablet_manager.cc:1398] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:00.540958 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.541496 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.551033 5548 tablet_service.cc:1467] Processing CreateTablet for tablet e3078f2bf97d42d4bde80035e04a47a6 (DEFAULT_TABLE table=default.Table_ad2e433251d046b589865483014c790e [id=88e838684d214496826cb851041ee2bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.551484 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8615f89e816e48fc8a058e5dc466918c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:00.583158 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3078f2bf97d42d4bde80035e04a47a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.585160 5839 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:12:00.586030 5654 leader_election.cc:341] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.586181 5654 leader_election.cc:302] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.586619 6376 raft_consensus.cc:2726] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:00.586742 5839 tablet_bootstrap.cc:654] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:00.589088 5378 leader_election.cc:341] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:00.589668 5378 leader_election.cc:341] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.551621 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 1d432a59bdd344ddaaf710de73b543e6 (DEFAULT_TABLE table=default.Table_22d1db9ba344416091aab96381212e34 [id=cb560b34838b48bfb987e60ea243f9e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.590857 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d432a59bdd344ddaaf710de73b543e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.489117 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 (DEFAULT_TABLE table=default.Table_a07279891494410096ed3d7da872b203 [id=76f5f0241cad4c09af24b15b3866bdd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.597474 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e3ea4658bb946439e4a50962444b3e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.601500 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe (DEFAULT_TABLE table=default.Table_fb9a13f3f676414d8c95abcc5889b14e [id=a311c58778dd486ea8a139917a690ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.601951 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173a2e351dea42e9938af5edb3c5c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.602589 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_48a8b33cbac2439689197979206121b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.603281 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48a8b33cbac2439689197979206121b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.606197 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe (DEFAULT_TABLE table=default.Table_fb9a13f3f676414d8c95abcc5889b14e [id=a311c58778dd486ea8a139917a690ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.606603 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173a2e351dea42e9938af5edb3c5c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.489383 5556 tablet_service.cc:1467] Processing CreateTablet for tablet d4dd191b107544d88e62f31c79e254b4 (DEFAULT_TABLE table=default.Table_d7cc7b9aea98415fbbfdfaf0cedab2de [id=8d27529a01c2475c862d26f344ec2728]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.567647 5554 tablet_service.cc:1467] Processing CreateTablet for tablet e7417948f54a490892e083caeddad6af (DEFAULT_TABLE table=default.Table_659bd38568664e459c30bf2ad102ff78 [id=50e6da1e1865462da3f718a2793c6119]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.606979 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7417948f54a490892e083caeddad6af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.608163 5838 tablet_replica.cc:330] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.608374 5838 raft_consensus.cc:2229] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.608618 5838 raft_consensus.cc:2258] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.609174 5838 ts_tablet_manager.cc:1875] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.573015 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.576560 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe (DEFAULT_TABLE table=default.Table_fb9a13f3f676414d8c95abcc5889b14e [id=a311c58778dd486ea8a139917a690ccb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.609992 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 173a2e351dea42e9938af5edb3c5c9fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.610678 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f (DEFAULT_TABLE table=default.Table_746e0a45757a4d1e898c096f11542589 [id=075087d8f9af4e189de5ff3e7004e78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.611049 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aca2077b76b49e3bf9daae57dc09b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.613826 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4dd191b107544d88e62f31c79e254b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.621471 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f (DEFAULT_TABLE table=default.Table_746e0a45757a4d1e898c096f11542589 [id=075087d8f9af4e189de5ff3e7004e78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.622560 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aca2077b76b49e3bf9daae57dc09b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.624845 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d4dd191b107544d88e62f31c79e254b4
I20240507 08:12:00.626868 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f (DEFAULT_TABLE table=default.Table_746e0a45757a4d1e898c096f11542589 [id=075087d8f9af4e189de5ff3e7004e78b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:00.627337 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aca2077b76b49e3bf9daae57dc09b8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:00.628065 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dd191b107544d88e62f31c79e254b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:00.628705 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d4dd191b107544d88e62f31c79e254b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4dd191b107544d88e62f31c79e254b4 already in progress: creating tablet
I20240507 08:12:00.628763 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dd191b107544d88e62f31c79e254b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.629334 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_22d1db9ba344416091aab96381212e34" } new_table_name: "default.Table_53fc83cdbcd7439ba2dd0ca50cae2094" modify_external_catalogs: true
W20240507 08:12:00.629396 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d4dd191b107544d88e62f31c79e254b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4dd191b107544d88e62f31c79e254b4 already in progress: creating tablet
I20240507 08:12:00.634722 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4867e1cca1634b4ab6e36f9ed3204e0a (table default.Table_1b3804a3ddb546bdb6a9d9813e2372d0 [id=d3239547346249628c8d25ec2637a7c9]) successfully deleted
I20240507 08:12:00.639930 5578 raft_consensus.cc:3037] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.645916 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet d4dd191b107544d88e62f31c79e254b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.653049 6441 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:00.669106 6441 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:00.679265 5843 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.679451 5843 ts_tablet_manager.cc:1367] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.120s user 0.002s sys 0.002s
I20240507 08:12:00.680166 5843 raft_consensus.cc:348] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.680397 5843 raft_consensus.cc:374] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.680469 5843 raft_consensus.cc:729] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.680686 5843 consensus_queue.cc:260] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.681916 6441 tablet.cc:1717] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Skipping requested alter to schema version 3, tablet already version 3
I20240507 08:12:00.691543 5995 raft_consensus.cc:482] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.691684 5995 raft_consensus.cc:504] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.692330 5995 leader_election.cc:288] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.693243 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde62374dc7f4a67a071b6b658e501c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:00.693373 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde62374dc7f4a67a071b6b658e501c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.693513 5718 raft_consensus.cc:2454] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:00.693989 5377 leader_election.cc:302] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:00.694406 5995 raft_consensus.cc:2781] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:00.694525 5995 raft_consensus.cc:482] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:00.694617 5995 raft_consensus.cc:3037] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:00.695475 5378 leader_election.cc:341] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.697319 5843 ts_tablet_manager.cc:1398] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240507 08:12:00.697451 5995 raft_consensus.cc:504] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.698491 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde62374dc7f4a67a071b6b658e501c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:00.699045 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde62374dc7f4a67a071b6b658e501c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:00.699254 5718 raft_consensus.cc:3037] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.700058 5843 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.701823 5843 tablet_bootstrap.cc:654] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:00.701978 5378 leader_election.cc:341] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.702374 5839 tablet_bootstrap.cc:492] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:00.702581 5718 raft_consensus.cc:2454] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:00.702615 5839 ts_tablet_manager.cc:1367] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.000s
I20240507 08:12:00.703117 5377 leader_election.cc:302] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.703397 6409 raft_consensus.cc:2781] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:00.703423 5839 raft_consensus.cc:348] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.703543 6409 raft_consensus.cc:686] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:00.703678 5839 raft_consensus.cc:374] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.703765 5839 raft_consensus.cc:729] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.703778 6409 consensus_queue.cc:237] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.703894 5995 leader_election.cc:288] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.704051 5839 consensus_queue.cc:260] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.711102 5839 ts_tablet_manager.cc:1398] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:00.717155 5839 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:00.719399 5839 tablet_bootstrap.cc:654] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.723150 5256 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_57ae6d7b295a4fb38adfb6579b01dc83 [id=b3fb019c81734b0f8108f607f4b15972] alter complete (version 2)
I20240507 08:12:00.723600 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 53fb38b567c5455ebc441e5f71db694f from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.723863 5843 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.724030 5843 ts_tablet_manager.cc:1367] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.004s
I20240507 08:12:00.724886 5843 raft_consensus.cc:348] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.725096 5843 raft_consensus.cc:374] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.725185 5843 raft_consensus.cc:729] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.725450 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 68f1eeccee8e49b3a1dbd1930c2d108a from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.725464 5843 consensus_queue.cc:260] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.729472 5838 ts_tablet_manager.cc:1888] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.729967 5843 ts_tablet_manager.cc:1398] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240507 08:12:00.731364 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.731616 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.731923 5837 tablet_replica.cc:330] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.732111 5837 raft_consensus.cc:2229] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.732301 5837 raft_consensus.cc:2258] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.732975 5843 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.733247 5837 ts_tablet_manager.cc:1875] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.734709 5843 tablet_bootstrap.cc:654] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.739114 5839 tablet_bootstrap.cc:492] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:00.739260 5839 ts_tablet_manager.cc:1367] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240507 08:12:00.740038 5839 raft_consensus.cc:348] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.740243 5839 raft_consensus.cc:374] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.740335 5838 log.cc:1196] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/0a0402c6efc8412aac0d21a545944073
I20240507 08:12:00.740442 5839 raft_consensus.cc:729] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.741973 5838 ts_tablet_manager.cc:1909] T 0a0402c6efc8412aac0d21a545944073 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.742630 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 0a0402c6efc8412aac0d21a545944073 (table default.Table_e85ccc16434742ac812466c315098b9a [id=42e0c3835e194cd6a54f6ad7d8e4f5ed]) successfully deleted
W20240507 08:12:00.743105 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 99e4514039304f0696c69a057db0ea6e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99e4514039304f0696c69a057db0ea6e already in progress: creating tablet
I20240507 08:12:00.744092 5837 ts_tablet_manager.cc:1888] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.744791 5837 log.cc:1196] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/6bf34089e5ec419f97c0da9ec42d7eec
I20240507 08:12:00.745216 5837 ts_tablet_manager.cc:1909] T 6bf34089e5ec419f97c0da9ec42d7eec P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.746274 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 6bf34089e5ec419f97c0da9ec42d7eec (table default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]) successfully deleted
I20240507 08:12:00.746619 5837 tablet_replica.cc:330] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.746752 5843 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.741384 5839 consensus_queue.cc:260] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.746961 5843 ts_tablet_manager.cc:1367] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240507 08:12:00.746909 5837 raft_consensus.cc:2229] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.747193 5837 raft_consensus.cc:2258] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.747740 5837 ts_tablet_manager.cc:1875] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.748581 5843 raft_consensus.cc:348] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.748869 5843 raft_consensus.cc:374] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.748953 5843 raft_consensus.cc:729] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.749302 5843 consensus_queue.cc:260] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.751081 5839 ts_tablet_manager.cc:1398] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240507 08:12:00.752022 5840 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.752214 5840 ts_tablet_manager.cc:1367] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.229s user 0.053s sys 0.053s
I20240507 08:12:00.753041 5840 raft_consensus.cc:348] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.753296 5840 raft_consensus.cc:374] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.753417 5840 raft_consensus.cc:729] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.753692 5840 consensus_queue.cc:260] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.754254 5843 ts_tablet_manager.cc:1398] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
W20240507 08:12:00.756215 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 975f22194aff40609a3cfa63b9b3aca2 consensus state has no leader
I20240507 08:12:00.756667 5840 ts_tablet_manager.cc:1398] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:12:00.756682 5843 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.756958 5840 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.758482 5843 tablet_bootstrap.cc:654] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.758759 5839 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:00.759186 5840 tablet_bootstrap.cc:654] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.759979 5838 tablet_replica.cc:330] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:00.760150 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3a065a4441f449a299515dda078f2d24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3a065a4441f449a299515dda078f2d24 already in progress: creating tablet
I20240507 08:12:00.760191 5838 raft_consensus.cc:2229] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.760377 5838 raft_consensus.cc:2258] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.761055 5838 ts_tablet_manager.cc:1875] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.764850 5837 ts_tablet_manager.cc:1888] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.765029 5837 log.cc:1196] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5554ebc621c24d459668f2cefc20b641
I20240507 08:12:00.765537 5837 ts_tablet_manager.cc:1909] T 5554ebc621c24d459668f2cefc20b641 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.769770 5843 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.769927 5843 ts_tablet_manager.cc:1367] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.001s
I20240507 08:12:00.770627 5843 raft_consensus.cc:348] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.771343 5843 raft_consensus.cc:374] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.771461 5843 raft_consensus.cc:729] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.772508 5843 consensus_queue.cc:260] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.773095 5840 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.773216 5840 ts_tablet_manager.cc:1367] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.006s
I20240507 08:12:00.774820 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5554ebc621c24d459668f2cefc20b641 (table default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]) successfully deleted
I20240507 08:12:00.775697 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bc5e572b5c614e0fbb526ab59a22254f" } new_table_name: "default.Table_82aad3e9a8064c75a66f0675c60c76e1" modify_external_catalogs: true
I20240507 08:12:00.775828 5843 ts_tablet_manager.cc:1398] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240507 08:12:00.776135 5843 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.778487 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_91931ba97b4c48ae92cb1ecb56850bd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.779021 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91931ba97b4c48ae92cb1ecb56850bd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.773970 5840 raft_consensus.cc:348] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.780179 5843 tablet_bootstrap.cc:654] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.780177 5840 raft_consensus.cc:374] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.780267 5840 raft_consensus.cc:729] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.780476 5839 tablet_bootstrap.cc:654] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.780529 5840 consensus_queue.cc:260] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.781875 5258 catalog_manager.cc:5484] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:00.782532 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:00.793536 5840 ts_tablet_manager.cc:1398] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240507 08:12:00.793637 5843 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.793807 5843 ts_tablet_manager.cc:1367] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20240507 08:12:00.793130 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_682aa156946f429890d5095fa23f097e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.794533 5843 raft_consensus.cc:348] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.794761 5843 raft_consensus.cc:374] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.794852 5843 raft_consensus.cc:729] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.795014 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet d9ff5e1b6a9a476c8295a0d0ee85aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.795095 5843 consensus_queue.cc:260] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:00.795814 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_682aa156946f429890d5095fa23f097e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.796643 5840 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.797065 5843 ts_tablet_manager.cc:1398] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:00.797379 5843 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.799044 5843 tablet_bootstrap.cc:654] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.801488 5840 tablet_bootstrap.cc:654] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.804720 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 385f4c1eb7f4493a924b813734ddcfdb
I20240507 08:12:00.806020 5838 ts_tablet_manager.cc:1888] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:00.806077 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.806195 5838 log.cc:1196] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/3a3f56ea5b7a43e4a89845da5a4d98f7
I20240507 08:12:00.806341 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.806624 5838 ts_tablet_manager.cc:1909] T 3a3f56ea5b7a43e4a89845da5a4d98f7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.807040 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.807610 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 3a3f56ea5b7a43e4a89845da5a4d98f7 (table default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]) successfully deleted
W20240507 08:12:00.807760 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 385f4c1eb7f4493a924b813734ddcfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 385f4c1eb7f4493a924b813734ddcfdb already in progress: creating tablet
W20240507 08:12:00.808127 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 385f4c1eb7f4493a924b813734ddcfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 385f4c1eb7f4493a924b813734ddcfdb already in progress: creating tablet
I20240507 08:12:00.809185 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5b058aadbe304a9b9c8ffd023c1a0470" } modify_external_catalogs: true
I20240507 08:12:00.809381 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5b058aadbe304a9b9c8ffd023c1a0470" } modify_external_catalogs: true
I20240507 08:12:00.810503 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c1becaf1c2694657b87263a7e941b119" } new_table_name: "default.Table_3da110b2f7014fdca83ebeeb4c70c3d1" modify_external_catalogs: true
I20240507 08:12:00.811260 5838 tablet_replica.cc:330] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.811455 5838 raft_consensus.cc:2229] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.811654 5838 raft_consensus.cc:2258] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:00.811770 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3c7f072f55a94cca938aa063637af9a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c7f072f55a94cca938aa063637af9a5 already in progress: creating tablet
I20240507 08:12:00.812251 5840 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.812274 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 60bc7bc06e494adcb6a8641b50b90e6c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.812472 5840 ts_tablet_manager.cc:1367] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.000s
I20240507 08:12:00.813151 5840 raft_consensus.cc:348] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.813416 5840 raft_consensus.cc:374] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.813565 5840 raft_consensus.cc:729] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.813871 5840 consensus_queue.cc:260] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.815640 5843 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.815707 5838 ts_tablet_manager.cc:1875] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.815788 5843 ts_tablet_manager.cc:1367] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240507 08:12:00.816527 5843 raft_consensus.cc:348] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.816709 5843 raft_consensus.cc:374] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.816805 5843 raft_consensus.cc:729] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.817049 5843 consensus_queue.cc:260] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.817417 5840 ts_tablet_manager.cc:1398] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:00.820199 5843 ts_tablet_manager.cc:1398] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:00.820255 5840 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.820493 5843 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.822011 5840 tablet_bootstrap.cc:654] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.823081 5843 tablet_bootstrap.cc:654] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.829887 5843 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.830092 5843 ts_tablet_manager.cc:1367] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240507 08:12:00.832191 5843 raft_consensus.cc:348] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.833096 5843 raft_consensus.cc:374] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.833823 5843 raft_consensus.cc:729] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.833736 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f8765d860074430f9fb9ecfba92d9dd7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.834100 5843 consensus_queue.cc:260] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:00.834296 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8765d860074430f9fb9ecfba92d9dd7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.834565 5843 ts_tablet_manager.cc:1398] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240507 08:12:00.834906 5843 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:00.835542 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d432a59bdd344ddaaf710de73b543e6 consensus state has no leader
I20240507 08:12:00.835832 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 270a1fd15925416483c211021549315c from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.836364 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dd26080257a24a50932dc596426fc19a" } modify_external_catalogs: true
I20240507 08:12:00.836532 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dd26080257a24a50932dc596426fc19a" } modify_external_catalogs: true
I20240507 08:12:00.837085 5843 tablet_bootstrap.cc:654] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.839784 5840 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.839951 5840 ts_tablet_manager.cc:1367] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.002s
I20240507 08:12:00.840730 5840 raft_consensus.cc:348] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.840950 5840 raft_consensus.cc:374] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.841013 5840 raft_consensus.cc:729] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.841285 5840 consensus_queue.cc:260] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.844367 5840 ts_tablet_manager.cc:1398] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:00.844655 5840 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:00.846405 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 093fb3986a674b93a435408c7ec85500 consensus state has no leader
I20240507 08:12:00.847714 5840 tablet_bootstrap.cc:654] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.850196 5838 ts_tablet_manager.cc:1888] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.850355 5838 log.cc:1196] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f49dd4772f8e46478638b29c009c83ea
I20240507 08:12:00.850786 5838 ts_tablet_manager.cc:1909] T f49dd4772f8e46478638b29c009c83ea P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.851971 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f49dd4772f8e46478638b29c009c83ea (table default.Table_866be50aa9504f60904fd1dc4d383a17 [id=eb99368bad8d45e5bad902171f4da9b8]) successfully deleted
I20240507 08:12:00.855131 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_336e8bfc36dd4d50a3a52a778ef1f102" } new_table_name: "default.Table_59179182d23d4f1f9e0311758a287f76" modify_external_catalogs: true
I20240507 08:12:00.875270 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet c84a2018f1c442e2a4dfc1c6b9c5ceef from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.875991 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 68f1eeccee8e49b3a1dbd1930c2d108a
I20240507 08:12:00.876518 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 68f1eeccee8e49b3a1dbd1930c2d108a deleted and replaced by tablet 128c2034d917473eb5c496319fdc9625
I20240507 08:12:00.879330 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a with delete_type TABLET_DATA_DELETED (replaced by tablet 128c2034d917473eb5c496319fdc9625) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.884217 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.884461 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.884622 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.884783 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a with delete_type TABLET_DATA_DELETED (replaced by tablet 128c2034d917473eb5c496319fdc9625) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.884987 5841 tablet_replica.cc:330] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.885174 5841 raft_consensus.cc:2229] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.885363 5841 raft_consensus.cc:2258] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.885937 5841 ts_tablet_manager.cc:1875] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.891191 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 68f1eeccee8e49b3a1dbd1930c2d108a with delete_type TABLET_DATA_DELETED (replaced by tablet 128c2034d917473eb5c496319fdc9625) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.891453 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.891656 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.891749 5837 tablet_replica.cc:330] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.891953 5837 raft_consensus.cc:2229] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.892117 5837 raft_consensus.cc:2258] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:00.892612 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 68f1eeccee8e49b3a1dbd1930c2d108a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f1eeccee8e49b3a1dbd1930c2d108a already in progress: creating tablet
I20240507 08:12:00.892949 5837 ts_tablet_manager.cc:1875] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:00.894079 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd8fc25f0dff4f3d9f9aae7d2694aa9a already in progress: creating tablet
W20240507 08:12:00.894654 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d4dd191b107544d88e62f31c79e254b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4dd191b107544d88e62f31c79e254b4 already in progress: creating tablet
I20240507 08:12:00.895191 5838 tablet_replica.cc:330] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.895396 5838 raft_consensus.cc:2229] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.895550 5838 raft_consensus.cc:2258] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.896306 5838 ts_tablet_manager.cc:1875] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.898941 5837 ts_tablet_manager.cc:1888] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.899060 5837 log.cc:1196] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/bde62374dc7f4a67a071b6b658e501c4
I20240507 08:12:00.899377 5837 ts_tablet_manager.cc:1909] T bde62374dc7f4a67a071b6b658e501c4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.900200 5839 tablet_bootstrap.cc:492] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:00.900238 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet bde62374dc7f4a67a071b6b658e501c4 (table default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3 [id=d0a57f2571ca478084824a44922ac99b]) successfully deleted
I20240507 08:12:00.900446 5839 ts_tablet_manager.cc:1367] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.142s user 0.002s sys 0.014s
I20240507 08:12:00.900625 5837 tablet_replica.cc:330] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.900858 5837 raft_consensus.cc:2229] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.901047 5837 raft_consensus.cc:2258] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.901501 5839 raft_consensus.cc:348] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.901773 5839 raft_consensus.cc:374] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.901832 5837 ts_tablet_manager.cc:1875] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.901894 5839 raft_consensus.cc:729] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.902338 5839 consensus_queue.cc:260] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.905522 5839 ts_tablet_manager.cc:1398] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:00.905743 5840 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:00.905923 5840 ts_tablet_manager.cc:1367] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.020s
I20240507 08:12:00.908694 5838 ts_tablet_manager.cc:1888] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.908844 5838 log.cc:1196] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d9ff5e1b6a9a476c8295a0d0ee85aecc
I20240507 08:12:00.909173 5838 ts_tablet_manager.cc:1909] T d9ff5e1b6a9a476c8295a0d0ee85aecc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.909968 5840 raft_consensus.cc:348] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.910825 5840 raft_consensus.cc:374] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.911031 5840 raft_consensus.cc:729] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.911378 5840 consensus_queue.cc:260] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.913554 5840 ts_tablet_manager.cc:1398] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:00.914206 5840 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:00.914641 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d9ff5e1b6a9a476c8295a0d0ee85aecc (table default.Table_a1158486249b4ac59cd8e9db4042a5d2 [id=7869057b367543fb83185721064121a9]) successfully deleted
W20240507 08:12:00.915249 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 385f4c1eb7f4493a924b813734ddcfdb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 385f4c1eb7f4493a924b813734ddcfdb already in progress: creating tablet
I20240507 08:12:00.915550 5838 tablet_replica.cc:330] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.915719 5838 raft_consensus.cc:2229] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.915886 5838 raft_consensus.cc:2258] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.917814 6409 raft_consensus.cc:482] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:00.917881 5840 tablet_bootstrap.cc:654] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.917987 6409 raft_consensus.cc:504] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.918689 5839 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:00.918674 6409 leader_election.cc:288] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.919836 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:00.920014 5718 raft_consensus.cc:2454] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:00.920344 5377 leader_election.cc:302] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:00.920707 6409 raft_consensus.cc:2781] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:00.920809 6409 raft_consensus.cc:482] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:00.920897 6409 raft_consensus.cc:3037] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.920972 5837 ts_tablet_manager.cc:1888] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.920919 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:00.921082 5837 log.cc:1196] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/191adef34efa4c3999d4642a80177294
I20240507 08:12:00.921290 5838 ts_tablet_manager.cc:1875] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.921314 5841 ts_tablet_manager.cc:1888] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.921430 5841 log.cc:1196] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/34bebec088a34ebbb06d2e707d54da4c
I20240507 08:12:00.921475 5837 ts_tablet_manager.cc:1909] T 191adef34efa4c3999d4642a80177294 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.922502 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 191adef34efa4c3999d4642a80177294 (table default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]) successfully deleted
I20240507 08:12:00.922902 5841 ts_tablet_manager.cc:1909] T 34bebec088a34ebbb06d2e707d54da4c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:00.923908 6409 raft_consensus.cc:504] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:00.924974 5378 leader_election.cc:341] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.925047 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:00.925227 5718 raft_consensus.cc:3037] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:00.925838 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba7c1aaf55d74c979ec1fc0a57f15f44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:00.926465 5378 leader_election.cc:341] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:00.926689 6409 leader_election.cc:288] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:00.928169 5718 raft_consensus.cc:2454] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:00.928361 5838 ts_tablet_manager.cc:1888] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.928462 5838 log.cc:1196] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/60bc7bc06e494adcb6a8641b50b90e6c
I20240507 08:12:00.928791 5838 ts_tablet_manager.cc:1909] T 60bc7bc06e494adcb6a8641b50b90e6c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:00.928755 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 34bebec088a34ebbb06d2e707d54da4c (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
I20240507 08:12:00.929384 5377 leader_election.cc:302] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:00.929515 5841 tablet_replica.cc:330] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:00.929728 5841 raft_consensus.cc:2229] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:00.929847 6409 raft_consensus.cc:2781] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:00.930486 6409 raft_consensus.cc:686] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:00.930569 5839 tablet_bootstrap.cc:654] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.930727 6409 consensus_queue.cc:237] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.937141 5841 pending_rounds.cc:62] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:00.937279 5841 pending_rounds.cc:68] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924756805009408 op_type: NO_OP noop_request { }
I20240507 08:12:00.937456 5841 raft_consensus.cc:2866] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:00.937660 5841 raft_consensus.cc:2258] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.938694 5841 ts_tablet_manager.cc:1875] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.941429 5839 tablet_bootstrap.cc:492] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:00.941673 5839 ts_tablet_manager.cc:1367] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.000s
I20240507 08:12:00.942668 5839 raft_consensus.cc:348] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.942981 5839 raft_consensus.cc:374] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.943104 5839 raft_consensus.cc:729] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.943411 5839 consensus_queue.cc:260] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.944413 5843 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.944557 5843 ts_tablet_manager.cc:1367] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.110s user 0.003s sys 0.032s
I20240507 08:12:00.945274 5843 raft_consensus.cc:348] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.945464 5843 raft_consensus.cc:374] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.945545 5843 raft_consensus.cc:729] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.945762 5843 consensus_queue.cc:260] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:00.946383 5843 ts_tablet_manager.cc:1398] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:00.946672 5843 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.947368 5839 ts_tablet_manager.cc:1398] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:12:00.947567 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fa1ba71ce9b5486f98e731e1ede13b36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.948223 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa1ba71ce9b5486f98e731e1ede13b36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.948295 5843 tablet_bootstrap.cc:654] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.949725 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f7557ddeec745f59ac90d4085a9015c" } modify_external_catalogs: true
I20240507 08:12:00.949920 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f7557ddeec745f59ac90d4085a9015c" } modify_external_catalogs: true
I20240507 08:12:00.950163 5839 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:00.952059 5839 tablet_bootstrap.cc:654] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.952133 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d2fbb40d8c9e405381a573eea7a1d5d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.952567 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2fbb40d8c9e405381a573eea7a1d5d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.953280 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 60bc7bc06e494adcb6a8641b50b90e6c (table default.Table_60d0f9c31e13466793596c325f991a88 [id=9245b95f8cc14f3898c4bdcb421954e2]) successfully deleted
W20240507 08:12:00.954030 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 68f1eeccee8e49b3a1dbd1930c2d108a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f1eeccee8e49b3a1dbd1930c2d108a already in progress: creating tablet
I20240507 08:12:00.959120 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.959858 5837 tablet_replica.cc:330] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.960052 5837 raft_consensus.cc:2229] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.960240 5837 raft_consensus.cc:2258] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.960682 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.960871 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.961378 5837 ts_tablet_manager.cc:1875] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.965059 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6450d3d2e7924ad2b72e8bd91378282b
W20240507 08:12:00.965770 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 279e95d9804c44eaa2c23a4ec02c84ff consensus state has no leader
I20240507 08:12:00.965711 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:00.966380 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6450d3d2e7924ad2b72e8bd91378282b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6450d3d2e7924ad2b72e8bd91378282b already in progress: creating tablet
I20240507 08:12:00.967062 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 1cd02a49359743c4b4e540ab0338d8ed from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:00.969666 5837 ts_tablet_manager.cc:1888] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.969811 5837 log.cc:1196] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:00.969889 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:00.969916 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8083493fc8df43ee8867ad0dca71c295"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:00.970230 5837 ts_tablet_manager.cc:1909] T d537bed29a2f47058255d2530b5c0d23 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:00.970435 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8083493fc8df43ee8867ad0dca71c295 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.971529 5837 tablet_replica.cc:330] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.971800 5837 raft_consensus.cc:2229] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.971974 5837 raft_consensus.cc:2258] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.972059 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d537bed29a2f47058255d2530b5c0d23 (table default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]) successfully deleted
I20240507 08:12:00.972409 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0fafe65af5f34be4b333cb6baa327f94" } modify_external_catalogs: true
I20240507 08:12:00.972554 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 6450d3d2e7924ad2b72e8bd91378282b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.972596 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0fafe65af5f34be4b333cb6baa327f94" } modify_external_catalogs: true
I20240507 08:12:00.972803 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:00.974594 5843 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.974803 5843 ts_tablet_manager.cc:1367] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.007s
I20240507 08:12:00.974876 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e7d9305c31714aba91c87411c2855c6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:00.975394 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d9305c31714aba91c87411c2855c6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:00.975770 5843 raft_consensus.cc:348] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.976042 5843 raft_consensus.cc:374] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.976122 5843 raft_consensus.cc:729] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:00.976967 5837 ts_tablet_manager.cc:1875] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.977955 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:00.978446 5838 tablet_replica.cc:330] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:00.978662 5838 raft_consensus.cc:2229] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:00.978860 5838 raft_consensus.cc:2258] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:00.980484 5843 consensus_queue.cc:260] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.981597 5843 ts_tablet_manager.cc:1398] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240507 08:12:00.981981 5843 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:00.987282 5843 tablet_bootstrap.cc:654] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:00.991222 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_225c3dd465f0444a9688b5c17d613ad5" } new_table_name: "default.Table_0b39549076654463a9689c30a3cb49b1" modify_external_catalogs: true
I20240507 08:12:00.991654 5837 ts_tablet_manager.cc:1888] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:00.991823 5837 log.cc:1196] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:12:00.992201 5838 ts_tablet_manager.cc:1875] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:00.992239 5837 ts_tablet_manager.cc:1909] T e1f9a56a95074e67bb33ceee1af48a18 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:00.993152 5837 tablet_replica.cc:330] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:00.993299 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e1f9a56a95074e67bb33ceee1af48a18 (table default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]) successfully deleted
I20240507 08:12:00.994980 5843 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:00.995124 5843 ts_tablet_manager.cc:1367] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240507 08:12:00.995844 5843 raft_consensus.cc:348] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.996059 5843 raft_consensus.cc:374] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:00.996162 5843 raft_consensus.cc:729] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:00.996310 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55a535a1039d43b9b0bf2c61304c2d01 consensus state has no leader
I20240507 08:12:00.998178 5843 consensus_queue.cc:260] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:00.998744 5843 ts_tablet_manager.cc:1398] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:00.999063 5843 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:01.000716 5843 tablet_bootstrap.cc:654] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.001819 5838 ts_tablet_manager.cc:1888] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.003001 5838 log.cc:1196] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/6bf34089e5ec419f97c0da9ec42d7eec
I20240507 08:12:01.003374 5837 raft_consensus.cc:2229] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.003471 5838 ts_tablet_manager.cc:1909] T 6bf34089e5ec419f97c0da9ec42d7eec P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:01.004143 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 6bf34089e5ec419f97c0da9ec42d7eec (table default.Table_af2b4d29f1e64f2d8e48d5b37c48318a [id=9b02e12d05a348f98b09a44e17d044f5]) successfully deleted
I20240507 08:12:01.003979 5837 raft_consensus.cc:2258] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.005280 5837 ts_tablet_manager.cc:1875] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.008730 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d23b6a487a6a4a20876771868a0d439a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.009560 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d23b6a487a6a4a20876771868a0d439a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.009969 5843 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:01.010134 5843 ts_tablet_manager.cc:1367] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240507 08:12:01.010596 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 6bf34089e5ec419f97c0da9ec42d7eec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.010797 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f5d566695d94c708e90ee30254084a1 with delete_type TABLET_DATA_DELETED (replaced by tablet 12e85df8b900468ba2584fd8f2e9bb2d) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.010917 5843 raft_consensus.cc:348] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.011126 5843 raft_consensus.cc:374] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.011199 5843 raft_consensus.cc:729] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:01.012795 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6bf34089e5ec419f97c0da9ec42d7eec because the tablet was not found. No further retry: Not found: Tablet not found: 6bf34089e5ec419f97c0da9ec42d7eec
I20240507 08:12:01.013140 5838 tablet_replica.cc:330] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:01.013957 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_109dc998f0b24d49824e20c1693703b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.014431 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_109dc998f0b24d49824e20c1693703b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.015972 5837 ts_tablet_manager.cc:1888] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.017081 5838 raft_consensus.cc:2229] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.017305 5838 raft_consensus.cc:2258] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.017726 5843 consensus_queue.cc:260] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.018704 5843 ts_tablet_manager.cc:1398] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:01.019460 5843 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:01.020081 5838 ts_tablet_manager.cc:1875] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.022307 5843 tablet_bootstrap.cc:654] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.025214 5837 log.cc:1196] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/072702ef2a6d43248132da6aa3a2d493
I20240507 08:12:01.025947 5837 ts_tablet_manager.cc:1909] T 072702ef2a6d43248132da6aa3a2d493 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:01.025933 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d3d5d2e042244132918d76affc48a306" } modify_external_catalogs: true
I20240507 08:12:01.026160 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d3d5d2e042244132918d76affc48a306" } modify_external_catalogs: true
I20240507 08:12:01.028034 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 072702ef2a6d43248132da6aa3a2d493 (table default.Table_c6bf901b2c4a491e8aa4c357670a5d08 [id=3d99e1ae7dc34ac49843ce5f159f891e]) successfully deleted
I20240507 08:12:01.029789 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b8ebbcf0d5a54d63a96a9374e839af83
W20240507 08:12:01.030220 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6450d3d2e7924ad2b72e8bd91378282b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6450d3d2e7924ad2b72e8bd91378282b already in progress: creating tablet
I20240507 08:12:01.030880 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.031440 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.031853 5837 tablet_replica.cc:330] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:01.032009 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.032074 5837 raft_consensus.cc:2229] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.032229 5837 raft_consensus.cc:2258] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.033339 5843 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:01.033510 5843 ts_tablet_manager.cc:1367] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20240507 08:12:01.034749 5837 ts_tablet_manager.cc:1875] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.035715 5843 raft_consensus.cc:348] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.039132 5843 raft_consensus.cc:374] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.039403 5843 raft_consensus.cc:729] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.040355 5843 consensus_queue.cc:260] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.042026 5843 ts_tablet_manager.cc:1398] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240507 08:12:01.046211 5843 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:01.048229 5843 tablet_bootstrap.cc:654] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.053169 5837 ts_tablet_manager.cc:1888] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.054045 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8c0ea847a8d142b3a28ac68403f103ab" } new_table_name: "default.Table_1682cef551e4416a88a682fe5c7f4081" modify_external_catalogs: true
I20240507 08:12:01.055240 5837 log.cc:1196] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5f2b43daab6b4deeac94fc6f1ff34a76
I20240507 08:12:01.056066 5837 ts_tablet_manager.cc:1909] T 5f2b43daab6b4deeac94fc6f1ff34a76 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:01.056968 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2717b2d928674c17809ef4b0141e352e consensus state has no leader
I20240507 08:12:01.059129 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2fbb40d8c9e405381a573eea7a1d5d6" } new_table_name: "default.Table_cf9647a579774e5ebc3de8cc3754fbef" modify_external_catalogs: true
I20240507 08:12:01.060700 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (table default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]) successfully deleted
I20240507 08:12:01.060531 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_20e0ec532f1549dd9c1df7c636346293"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:01.060827 6409 raft_consensus.cc:482] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:01.061034 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b8ebbcf0d5a54d63a96a9374e839af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8ebbcf0d5a54d63a96a9374e839af83 already in progress: creating tablet
I20240507 08:12:01.060971 6409 raft_consensus.cc:504] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:01.061203 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20e0ec532f1549dd9c1df7c636346293 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.062027 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1f9a56a95074e67bb33ceee1af48a18" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.062788 5378 leader_election.cc:341] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.063360 6409 leader_election.cc:288] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:01.063838 5377 leader_election.cc:341] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:12:01.063985 5377 leader_election.cc:302] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.064429 6409 raft_consensus.cc:2726] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:01.067600 5843 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:01.067765 5843 ts_tablet_manager.cc:1367] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20240507 08:12:01.068627 5843 raft_consensus.cc:348] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.068852 5843 raft_consensus.cc:374] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.068935 5843 raft_consensus.cc:729] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.069221 5843 consensus_queue.cc:260] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.069866 5843 ts_tablet_manager.cc:1398] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20240507 08:12:01.071520 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dc49fd6841c420288aa1df3950a088b consensus state has no leader
I20240507 08:12:01.072901 5843 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:01.075031 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 690b258a9b564ed1b07aef29a774b749 has no consensus state
I20240507 08:12:01.076203 5843 tablet_bootstrap.cc:654] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.080716 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0f91e3e5674d49efa25f246c35d4f912"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.081270 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f91e3e5674d49efa25f246c35d4f912 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.114728 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet e182d186e6fe4e388a68c61ccf159f8d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:01.115033 5841 ts_tablet_manager.cc:1888] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:01.115165 5841 log.cc:1196] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/bde62374dc7f4a67a071b6b658e501c4
I20240507 08:12:01.115633 5841 ts_tablet_manager.cc:1909] T bde62374dc7f4a67a071b6b658e501c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:01.116720 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet bde62374dc7f4a67a071b6b658e501c4 (table default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3 [id=d0a57f2571ca478084824a44922ac99b]) successfully deleted
I20240507 08:12:01.117590 6409 raft_consensus.cc:482] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.117735 6409 raft_consensus.cc:504] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.117986 5840 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.118140 5840 ts_tablet_manager.cc:1367] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.204s user 0.031s sys 0.037s
I20240507 08:12:01.118423 6409 leader_election.cc:288] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:01.118856 5840 raft_consensus.cc:348] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.119076 5840 raft_consensus.cc:374] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.119132 5840 raft_consensus.cc:729] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.119187 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d537bed29a2f47058255d2530b5c0d23" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:01.119339 5840 consensus_queue.cc:260] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:01.120263 5378 leader_election.cc:341] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:01.123780 5377 leader_election.cc:341] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:01.124526 5840 ts_tablet_manager.cc:1398] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:12:01.125924 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.126204 5837 tablet_replica.cc:330] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:01.126394 5837 raft_consensus.cc:2229] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.126542 5837 raft_consensus.cc:2258] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.127444 5083 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskrbFV97/build/asan/bin/kudu pid 5940
I20240507 08:12:01.128676 5837 ts_tablet_manager.cc:1875] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.131482 5839 tablet_bootstrap.cc:492] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.131711 5839 ts_tablet_manager.cc:1367] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.182s user 0.016s sys 0.006s
I20240507 08:12:01.132699 5839 raft_consensus.cc:348] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.132995 5839 raft_consensus.cc:374] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.133092 5839 raft_consensus.cc:729] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.134438 6376 raft_consensus.cc:482] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.133426 5839 consensus_queue.cc:260] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.134593 6376 raft_consensus.cc:504] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.135267 6376 leader_election.cc:288] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.135905 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.136584 5654 leader_election.cc:341] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.137362 5837 ts_tablet_manager.cc:1888] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.137562 5837 log.cc:1196] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:12:01.137625 5840 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.138048 5837 ts_tablet_manager.cc:1909] T 907b5cb0e7a54903abe0b0d0e8d617dc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:01.139009 5377 leader_election.cc:302] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.139775 6409 raft_consensus.cc:2726] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:01.140058 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:01.140345 5444 raft_consensus.cc:2454] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:01.140790 5840 tablet_bootstrap.cc:654] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.141178 5656 leader_election.cc:302] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.141301 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 907b5cb0e7a54903abe0b0d0e8d617dc (table default.Table_74a9e58e257645d6b826cab9e4134f48 [id=0e277dd692e842b28638b8675da40ec9]) successfully deleted
I20240507 08:12:01.141566 6376 raft_consensus.cc:2781] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:01.141682 6376 raft_consensus.cc:482] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:01.141752 6376 raft_consensus.cc:3037] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.143718 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e3ffaf1781e4b108f305346476905be with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.144984 5839 ts_tablet_manager.cc:1398] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20240507 08:12:01.145057 6376 raft_consensus.cc:504] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.147285 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:01.147907 5654 leader_election.cc:341] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.148247 5841 tablet_replica.cc:330] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:01.148488 5841 raft_consensus.cc:2229] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.148746 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:01.148952 5444 raft_consensus.cc:3037] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.149173 5841 raft_consensus.cc:2258] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.151526 5839 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:01.151846 5444 raft_consensus.cc:2454] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:01.152880 5656 leader_election.cc:302] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.153457 5839 tablet_bootstrap.cc:654] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.154618 6376 leader_election.cc:288] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.155076 6376 raft_consensus.cc:2781] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:01.155220 5840 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.155555 5840 ts_tablet_manager.cc:1367] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.005s
I20240507 08:12:01.155606 6376 raft_consensus.cc:686] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:01.156250 5838 ts_tablet_manager.cc:1888] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.156380 5838 log.cc:1196] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1f5d566695d94c708e90ee30254084a1
I20240507 08:12:01.156338 5840 raft_consensus.cc:348] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.156572 5840 raft_consensus.cc:374] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.156672 5840 raft_consensus.cc:729] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.156752 5838 ts_tablet_manager.cc:1909] T 1f5d566695d94c708e90ee30254084a1 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:01.156260 6376 consensus_queue.cc:237] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.157373 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1f5d566695d94c708e90ee30254084a1 (table default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]) successfully deleted
I20240507 08:12:01.170444 5840 consensus_queue.cc:260] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.174655 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:01.176180 5840 ts_tablet_manager.cc:1398] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240507 08:12:01.176795 6482 raft_consensus.cc:482] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.177004 6482 raft_consensus.cc:504] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.177783 6482 leader_election.cc:288] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.178169 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.178704 5654 leader_election.cc:341] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.179936 5841 ts_tablet_manager.cc:1875] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.180097 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.180331 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.180521 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.180733 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.183384 5840 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.185012 5840 tablet_bootstrap.cc:654] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.185169 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 (DEFAULT_TABLE table=default.Table_a3e97a9c990549b880b71e973f807616 [id=4ce01e2ba9064412836c1adf33dd0e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.186290 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 (DEFAULT_TABLE table=default.Table_0bba3367e45542dcb9de73f0f9a05a6f [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.186868 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 7f18168519594979b02d651f093b2115 (DEFAULT_TABLE table=default.Table_4d08d2f517e6420aa76bcf5ec7928959 [id=7220df57a45f45bea3d887716f9cdec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.188122 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 (DEFAULT_TABLE table=default.Table_a3e97a9c990549b880b71e973f807616 [id=4ce01e2ba9064412836c1adf33dd0e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.188521 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f3b6d22f7840e7b43da7a52c25a810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.190666 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 (DEFAULT_TABLE table=default.Table_0bba3367e45542dcb9de73f0f9a05a6f [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.191030 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a23d112fcd4ea1b0c2f86133e82e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.193454 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 (DEFAULT_TABLE table=default.Table_e292ef7310c5489f8b3c60c61ab75b92 [id=21e5f8b3a5b648529837a27fbc86fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.193907 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 7f18168519594979b02d651f093b2115 (DEFAULT_TABLE table=default.Table_4d08d2f517e6420aa76bcf5ec7928959 [id=7220df57a45f45bea3d887716f9cdec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.195180 5421 tablet_service.cc:1467] Processing CreateTablet for tablet e941e3377db04a6388ea05ca49010874 (DEFAULT_TABLE table=default.Table_981d958d70ea464e8dd9a203a2374687 [id=553b860f0d114e529385ecbd792d2ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.195659 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f3b6d22f7840e7b43da7a52c25a810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.196278 5840 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.196434 5840 ts_tablet_manager.cc:1367] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240507 08:12:01.198201 5840 raft_consensus.cc:348] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.198405 5840 raft_consensus.cc:374] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.198463 5840 raft_consensus.cc:729] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.198688 5840 consensus_queue.cc:260] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.199450 5840 ts_tablet_manager.cc:1398] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240507 08:12:01.199748 5840 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.201903 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f18168519594979b02d651f093b2115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.205075 5840 tablet_bootstrap.cc:654] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.205729 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a23d112fcd4ea1b0c2f86133e82e32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.210158 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e1b850d9c6864653b0ca7439322d02a7" } modify_external_catalogs: true
I20240507 08:12:01.210431 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e1b850d9c6864653b0ca7439322d02a7" } modify_external_catalogs: true
I20240507 08:12:01.211685 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:01.212082 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet b2de528d9fc046f183f70388cffab6c8 (table default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a]). Expected version 2 got 0
I20240507 08:12:01.212044 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_54232a23a7cc453290c1ee7b44906b9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:01.212365 5840 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.212494 5840 ts_tablet_manager.cc:1367] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.004s
I20240507 08:12:01.212316 5258 catalog_manager.cc:5484] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
W20240507 08:12:01.212627 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54232a23a7cc453290c1ee7b44906b9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.213330 5840 raft_consensus.cc:348] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.213524 5840 raft_consensus.cc:374] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.213639 5840 raft_consensus.cc:729] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.213630 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8f7c06393ac54a15842f97319969eff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:01.213824 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ad2e433251d046b589865483014c790e" } new_table_name: "default.Table_be5564d556f345cc803276f9e26bf52d" modify_external_catalogs: true
W20240507 08:12:01.214222 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f7c06393ac54a15842f97319969eff1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.213963 5840 consensus_queue.cc:260] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.193914 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e44c3f6f96e417b92878d8ba4ff88a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.215024 5840 ts_tablet_manager.cc:1398] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:01.215288 5840 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.216856 5840 tablet_bootstrap.cc:654] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.217759 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f18168519594979b02d651f093b2115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.218394 5841 ts_tablet_manager.cc:1888] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.218555 5841 log.cc:1196] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5554ebc621c24d459668f2cefc20b641
I20240507 08:12:01.218984 5841 ts_tablet_manager.cc:1909] T 5554ebc621c24d459668f2cefc20b641 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:01.223904 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 (DEFAULT_TABLE table=default.Table_e292ef7310c5489f8b3c60c61ab75b92 [id=21e5f8b3a5b648529837a27fbc86fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.224370 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e44c3f6f96e417b92878d8ba4ff88a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.229864 5554 tablet_service.cc:1467] Processing CreateTablet for tablet e941e3377db04a6388ea05ca49010874 (DEFAULT_TABLE table=default.Table_981d958d70ea464e8dd9a203a2374687 [id=553b860f0d114e529385ecbd792d2ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.230274 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e941e3377db04a6388ea05ca49010874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.233924 5560 tablet_service.cc:1467] Processing CreateTablet for tablet a6cef6772a0f432891b63dd745d57873 (DEFAULT_TABLE table=default.Table_bf6fa5a914c14039a8f8940d2197f70a [id=54a07cced9e44b7c9768a53f050f98d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.234997 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 43225934cef04109b3dbc057b230f032
I20240507 08:12:01.235741 5840 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.235935 5840 ts_tablet_manager.cc:1367] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20240507 08:12:01.236694 5840 raft_consensus.cc:348] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.236929 5840 raft_consensus.cc:374] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.237012 5840 raft_consensus.cc:729] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.237247 5840 consensus_queue.cc:260] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.237697 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b01d5fc4ce904601b66e4e880c65d768" } new_table_name: "default.Table_a2c072fbeb35480fa276c46cbc1c6aa7" modify_external_catalogs: true
I20240507 08:12:01.239840 5416 tablet_service.cc:1467] Processing CreateTablet for tablet b27f89f6db6746fb870dad2a3fb27457 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.240417 5426 tablet_service.cc:1467] Processing CreateTablet for tablet b1d5627f71694fb68a15fb186c9068ce (DEFAULT_TABLE table=default.Table_63548a17f66844648a2d71fb0c7d7175 [id=a3a382d2cd854dba89a7be0230de2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.240762 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 (DEFAULT_TABLE table=default.Table_5ee780b3d01a481b80949a3ade7e3cd3 [id=dd2fd8b019214415861bddf4401af755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.241173 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 (DEFAULT_TABLE table=default.Table_3634efacf6494a4eabe5a4d9451e64a6 [id=a702c877339444a6850eb21b28b8d11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.241751 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 (DEFAULT_TABLE table=default.Table_65b762c4d6ac480eb6fbc20827e51a8a [id=ca64c52f4d254b63a7aae611974721db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.242267 5415 tablet_service.cc:1467] Processing CreateTablet for tablet a6cef6772a0f432891b63dd745d57873 (DEFAULT_TABLE table=default.Table_bf6fa5a914c14039a8f8940d2197f70a [id=54a07cced9e44b7c9768a53f050f98d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.242686 5420 tablet_service.cc:1467] Processing CreateTablet for tablet b95711fb20684223a02b09114419ce9a (DEFAULT_TABLE table=default.Table_8e9ea4672e9e4edcab31a6d54cd933e3 [id=3579ae2e81de4fd5b396a6fec9b6bce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.243203 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5866ebb974ee6842d8c66671a8467 (DEFAULT_TABLE table=default.Table_fdbd31dead3043cba3f6dce0b222bd13 [id=f8265b6b16534ca2b33d414b396fbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.243759 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 234e582b211940aea3471e5accb9d016 (DEFAULT_TABLE table=default.Table_e3df44d5188247b6a317ba4185f0b745 [id=d3d7815d2d51403094275109e542bbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.244302 5414 tablet_service.cc:1467] Processing CreateTablet for tablet d1319db1a6ea41f1a4eecf02ff67d217 (DEFAULT_TABLE table=default.Table_15f91f4b2cc24f138a6758b2a182bee7 [id=8e4a0ff05fc9437fbdbcc5de53caa048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.244834 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 5481ddbcd31449c4b44afb87a97138bd (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.245383 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d (DEFAULT_TABLE table=default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.246044 5840 ts_tablet_manager.cc:1398] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:12:01.246393 5840 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.246392 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 43225934cef04109b3dbc057b230f032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.246984 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 7818e3f7c1234bf6827826172d1e6af1 (DEFAULT_TABLE table=default.Table_ee5c0609839348ad9fd8706992998487 [id=e293639f88e04fea8adb90e380bdfd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.247485 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:01.247640 5409 tablet_service.cc:1467] Processing CreateTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa (DEFAULT_TABLE table=default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.247721 5444 raft_consensus.cc:2454] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:01.248315 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e941e3377db04a6388ea05ca49010874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.250101 5840 tablet_bootstrap.cc:654] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.253983 5656 leader_election.cc:302] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.254520 6376 raft_consensus.cc:2781] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:01.254650 6376 raft_consensus.cc:482] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:01.254771 6376 raft_consensus.cc:3037] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.259033 5841 tablet_replica.cc:330] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:01.259248 5841 raft_consensus.cc:2229] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.259449 5841 raft_consensus.cc:2258] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.259909 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27f89f6db6746fb870dad2a3fb27457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.262167 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4e37b73134cf42ad84b6eec90c9b805f
I20240507 08:12:01.262871 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2961308b2bc546de99dfee745feced1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.263396 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2961308b2bc546de99dfee745feced1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.265604 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e182d186e6fe4e388a68c61ccf159f8d
I20240507 08:12:01.265908 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d5627f71694fb68a15fb186c9068ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.266058 5263 catalog_manager.cc:6554] ReplaceTablet: tablet e182d186e6fe4e388a68c61ccf159f8d deleted and replaced by tablet 2b14c80b96bb43088e1d1354a1dbdd31
I20240507 08:12:01.266384 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ad8e609d07646e49e01cc55ba8ee969" } modify_external_catalogs: true
I20240507 08:12:01.266568 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ad8e609d07646e49e01cc55ba8ee969" } modify_external_catalogs: true
I20240507 08:12:01.266788 5841 ts_tablet_manager.cc:1875] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.269922 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924d8d2fe2f148cba13f107e4c70fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.273917 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48c9720c1a4a8b849730f28f6e00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.280419 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 (DEFAULT_TABLE table=default.Table_a3e97a9c990549b880b71e973f807616 [id=4ce01e2ba9064412836c1adf33dd0e50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.280629 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5554ebc621c24d459668f2cefc20b641 (table default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]) successfully deleted
I20240507 08:12:01.280864 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62f3b6d22f7840e7b43da7a52c25a810. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.281011 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 (DEFAULT_TABLE table=default.Table_0bba3367e45542dcb9de73f0f9a05a6f [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:01.281193 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 68f1eeccee8e49b3a1dbd1930c2d108a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f1eeccee8e49b3a1dbd1930c2d108a already in progress: creating tablet
I20240507 08:12:01.281340 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 37a23d112fcd4ea1b0c2f86133e82e32. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.281916 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6450d3d2e7924ad2b72e8bd91378282b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6450d3d2e7924ad2b72e8bd91378282b already in progress: creating tablet
W20240507 08:12:01.282173 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b8ebbcf0d5a54d63a96a9374e839af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8ebbcf0d5a54d63a96a9374e839af83 already in progress: creating tablet
I20240507 08:12:01.278493 6409 raft_consensus.cc:482] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.285797 6409 raft_consensus.cc:504] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.285904 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 7f18168519594979b02d651f093b2115 (DEFAULT_TABLE table=default.Table_4d08d2f517e6420aa76bcf5ec7928959 [id=7220df57a45f45bea3d887716f9cdec0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.286146 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e941e3377db04a6388ea05ca49010874 (DEFAULT_TABLE table=default.Table_981d958d70ea464e8dd9a203a2374687 [id=553b860f0d114e529385ecbd792d2ae0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.286267 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f18168519594979b02d651f093b2115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.286453 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e941e3377db04a6388ea05ca49010874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.286481 6409 leader_election.cc:288] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:01.283739 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 124ea88554ec4182a6682457cd58dfe1 (DEFAULT_TABLE table=default.Table_b17576c2e21b405380ccce99ebbd7b7e [id=ff2e74d195d241e99f1a1e24945a0b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.277560 5841 ts_tablet_manager.cc:1888] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.287218 5841 log.cc:1196] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5f2b43daab6b4deeac94fc6f1ff34a76
I20240507 08:12:01.287398 5690 tablet_service.cc:1467] Processing CreateTablet for tablet a6cef6772a0f432891b63dd745d57873 (DEFAULT_TABLE table=default.Table_bf6fa5a914c14039a8f8940d2197f70a [id=54a07cced9e44b7c9768a53f050f98d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.287612 5841 ts_tablet_manager.cc:1909] T 5f2b43daab6b4deeac94fc6f1ff34a76 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:01.287752 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cef6772a0f432891b63dd745d57873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.289198 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 957775c01be94cc081db344c95a81310
I20240507 08:12:01.285904 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 (DEFAULT_TABLE table=default.Table_e292ef7310c5489f8b3c60c61ab75b92 [id=21e5f8b3a5b648529837a27fbc86fa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.290104 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e44c3f6f96e417b92878d8ba4ff88a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.290428 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b8935f7c20ed43a69e9cfcb28fe5d023
I20240507 08:12:01.304988 5257 catalog_manager.cc:5484] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:01.305753 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet b2de528d9fc046f183f70388cffab6c8 (table default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a]). Expected version 2 got 0
I20240507 08:12:01.308233 6376 raft_consensus.cc:504] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.309532 6376 leader_election.cc:288] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.310035 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:01.310281 5444 raft_consensus.cc:3037] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.318799 5689 tablet_service.cc:1467] Processing CreateTablet for tablet b27f89f6db6746fb870dad2a3fb27457 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.319296 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27f89f6db6746fb870dad2a3fb27457. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.321414 5258 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 169626f9a0bc4c6f9d8be11c0a802816 consensus state has no leader
I20240507 08:12:01.322382 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 (DEFAULT_TABLE table=default.Table_3634efacf6494a4eabe5a4d9451e64a6 [id=a702c877339444a6850eb21b28b8d11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.322788 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48c9720c1a4a8b849730f28f6e00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.322824 5684 tablet_service.cc:1467] Processing CreateTablet for tablet b95711fb20684223a02b09114419ce9a (DEFAULT_TABLE table=default.Table_8e9ea4672e9e4edcab31a6d54cd933e3 [id=3579ae2e81de4fd5b396a6fec9b6bce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.323122 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95711fb20684223a02b09114419ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.322404 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 (DEFAULT_TABLE table=default.Table_65b762c4d6ac480eb6fbc20827e51a8a [id=ca64c52f4d254b63a7aae611974721db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.323469 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9669c1169b5244218383f7aa7fbc3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.326023 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5866ebb974ee6842d8c66671a8467 (DEFAULT_TABLE table=default.Table_fdbd31dead3043cba3f6dce0b222bd13 [id=f8265b6b16534ca2b33d414b396fbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.326610 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5866ebb974ee6842d8c66671a8467. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.329052 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3078f2bf97d42d4bde80035e04a47a6 consensus state has no leader
I20240507 08:12:01.329885 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cef6772a0f432891b63dd745d57873. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.330509 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07a9324570984372af7b9574f0cf455a consensus state has no leader
I20240507 08:12:01.331508 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d67b71d4e235444db255a5638a555d43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.332047 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67b71d4e235444db255a5638a555d43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.333446 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6cdc86ef760942ebbb3794342c9ccac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.333988 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cdc86ef760942ebbb3794342c9ccac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.335170 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1462e4ac39014fb0aafe7485fe690e29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.335587 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1462e4ac39014fb0aafe7485fe690e29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.337224 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 04d5866ebb974ee6842d8c66671a8467 (DEFAULT_TABLE table=default.Table_fdbd31dead3043cba3f6dce0b222bd13 [id=f8265b6b16534ca2b33d414b396fbdf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.337409 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_329152daa8064589a74946557194168f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:01.337745 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5866ebb974ee6842d8c66671a8467. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.338135 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_329152daa8064589a74946557194168f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.340329 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8c20f0267d2b49e2a564ec31f3cd52a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.340871 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c20f0267d2b49e2a564ec31f3cd52a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.342612 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_75025136df1f453e92ffc67c475fbb56" } new_table_name: "default.Table_56975594d01a4159aeafef5258ff445d" modify_external_catalogs: true
I20240507 08:12:01.344429 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 (DEFAULT_TABLE table=default.Table_3634efacf6494a4eabe5a4d9451e64a6 [id=a702c877339444a6850eb21b28b8d11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.344918 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 (DEFAULT_TABLE table=default.Table_65b762c4d6ac480eb6fbc20827e51a8a [id=ca64c52f4d254b63a7aae611974721db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.345450 5553 tablet_service.cc:1467] Processing CreateTablet for tablet b1d5627f71694fb68a15fb186c9068ce (DEFAULT_TABLE table=default.Table_63548a17f66844648a2d71fb0c7d7175 [id=a3a382d2cd854dba89a7be0230de2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.346268 5548 tablet_service.cc:1467] Processing CreateTablet for tablet b27f89f6db6746fb870dad2a3fb27457 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.343940 5562 tablet_service.cc:1467] Processing CreateTablet for tablet b95711fb20684223a02b09114419ce9a (DEFAULT_TABLE table=default.Table_8e9ea4672e9e4edcab31a6d54cd933e3 [id=3579ae2e81de4fd5b396a6fec9b6bce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.347640 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 (DEFAULT_TABLE table=default.Table_5ee780b3d01a481b80949a3ade7e3cd3 [id=dd2fd8b019214415861bddf4401af755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.345217 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 234e582b211940aea3471e5accb9d016 (DEFAULT_TABLE table=default.Table_e3df44d5188247b6a317ba4185f0b745 [id=d3d7815d2d51403094275109e542bbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.353835 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f38a29b591904253b1165f8512dce2a7" } new_table_name: "default.Table_33e54141e4cb44c3974edcf81302267f" modify_external_catalogs: true
I20240507 08:12:01.355110 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 43225934cef04109b3dbc057b230f032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.355602 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 5481ddbcd31449c4b44afb87a97138bd (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.357012 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c48c9720c1a4a8b849730f28f6e00d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.358294 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6563c42d240b4e8685dfa4972f2b9924
I20240507 08:12:01.358546 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9669c1169b5244218383f7aa7fbc3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.359215 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ea88554ec4182a6682457cd58dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.359750 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a65a8e939f4b7d89d00720845ea1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.360211 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5481ddbcd31449c4b44afb87a97138bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.360720 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95711fb20684223a02b09114419ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.361196 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1319db1a6ea41f1a4eecf02ff67d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.361692 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04d5866ebb974ee6842d8c66671a8467. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.362156 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e582b211940aea3471e5accb9d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.362605 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e85df8b900468ba2584fd8f2e9bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.363035 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7818e3f7c1234bf6827826172d1e6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.363556 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6cef6772a0f432891b63dd745d57873. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.369941 5841 tablet_replica.cc:330] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:01.370283 5841 raft_consensus.cc:2229] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.370486 5841 raft_consensus.cc:2258] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.371167 5841 ts_tablet_manager.cc:1875] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.377480 5559 tablet_service.cc:1467] Processing CreateTablet for tablet d1319db1a6ea41f1a4eecf02ff67d217 (DEFAULT_TABLE table=default.Table_15f91f4b2cc24f138a6758b2a182bee7 [id=8e4a0ff05fc9437fbdbcc5de53caa048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.381768 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e582b211940aea3471e5accb9d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.381917 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (table default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]) successfully deleted
I20240507 08:12:01.384045 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 7818e3f7c1234bf6827826172d1e6af1 (DEFAULT_TABLE table=default.Table_ee5c0609839348ad9fd8706992998487 [id=e293639f88e04fea8adb90e380bdfd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.389371 5681 tablet_service.cc:1467] Processing CreateTablet for tablet d1319db1a6ea41f1a4eecf02ff67d217 (DEFAULT_TABLE table=default.Table_15f91f4b2cc24f138a6758b2a182bee7 [id=8e4a0ff05fc9437fbdbcc5de53caa048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.390231 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 234e582b211940aea3471e5accb9d016 (DEFAULT_TABLE table=default.Table_e3df44d5188247b6a317ba4185f0b745 [id=d3d7815d2d51403094275109e542bbc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.391983 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 234e582b211940aea3471e5accb9d016. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.397193 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1319db1a6ea41f1a4eecf02ff67d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.397173 5550 tablet_service.cc:1467] Processing CreateTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa (DEFAULT_TABLE table=default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.397596 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d (DEFAULT_TABLE table=default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.398094 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d5627f71694fb68a15fb186c9068ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.399067 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5481ddbcd31449c4b44afb87a97138bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.400126 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a65a8e939f4b7d89d00720845ea1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.408586 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1319db1a6ea41f1a4eecf02ff67d217. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.413653 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e85df8b900468ba2584fd8f2e9bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.418570 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7818e3f7c1234bf6827826172d1e6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.424460 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 124ea88554ec4182a6682457cd58dfe1 (DEFAULT_TABLE table=default.Table_b17576c2e21b405380ccce99ebbd7b7e [id=ff2e74d195d241e99f1a1e24945a0b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.424860 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924d8d2fe2f148cba13f107e4c70fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.425566 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b95711fb20684223a02b09114419ce9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.427170 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9669c1169b5244218383f7aa7fbc3b85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.429628 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ea88554ec4182a6682457cd58dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.434779 5444 raft_consensus.cc:2454] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:01.435457 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 5481ddbcd31449c4b44afb87a97138bd (DEFAULT_TABLE table=default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.435510 5656 leader_election.cc:302] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:01.439600 5841 ts_tablet_manager.cc:1888] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.439812 5841 log.cc:1196] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/916e33e40ef44d06a359b82d8fe48157
I20240507 08:12:01.440192 5843 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:01.440356 5841 ts_tablet_manager.cc:1909] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:01.441162 5841 tablet_replica.cc:330] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:01.441277 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 916e33e40ef44d06a359b82d8fe48157 (table default.Table_6ad0c511f95b4b09abad3bc4a13d1238 [id=14589abdd357409cbe9837a33208e00d]) successfully deleted
I20240507 08:12:01.440357 5843 ts_tablet_manager.cc:1367] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.368s user 0.034s sys 0.068s
I20240507 08:12:01.442348 5843 raft_consensus.cc:348] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.442544 5843 raft_consensus.cc:374] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.442636 5843 raft_consensus.cc:729] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.442759 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 (DEFAULT_TABLE table=default.Table_375f890255ed48659c41d4c11bb30578 [id=8029d1e576b048a39b876990392bafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.442898 5843 consensus_queue.cc:260] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.443362 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323b96ae7d1e471998a39f3be0857ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.451335 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5481ddbcd31449c4b44afb87a97138bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.453233 6376 raft_consensus.cc:2781] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:01.453953 6376 raft_consensus.cc:686] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:01.454890 6376 consensus_queue.cc:237] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.459800 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d (DEFAULT_TABLE table=default.Table_71341b71104946439f1c74a9dc7b9c3d [id=cff2cbef9d27459fb6e3d10e7205759c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.460289 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12e85df8b900468ba2584fd8f2e9bb2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.464517 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b27f89f6db6746fb870dad2a3fb27457. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.469398 6301 generic_service.cc:196] Checking for leaks (request via RPC)
I20240507 08:12:01.476342 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 124ea88554ec4182a6682457cd58dfe1 (DEFAULT_TABLE table=default.Table_b17576c2e21b405380ccce99ebbd7b7e [id=ff2e74d195d241e99f1a1e24945a0b3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.476815 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 89f80e643c59420a90d2d57ca0b703c7 (DEFAULT_TABLE table=default.Table_96bb14500b6740f7bc08506e19f12519 [id=19fa9ceb2fa1458cbbf10c878ffa1853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.476944 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ea88554ec4182a6682457cd58dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.477139 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f80e643c59420a90d2d57ca0b703c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.474202 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 7818e3f7c1234bf6827826172d1e6af1 (DEFAULT_TABLE table=default.Table_ee5c0609839348ad9fd8706992998487 [id=e293639f88e04fea8adb90e380bdfd9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.478351 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9324570984372af7b9574f0cf455a (DEFAULT_TABLE table=default.Table_b01d5fc4ce904601b66e4e880c65d768 [id=f9c03e081d704f0b9f0c60b97a2265c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.478863 5683 tablet_service.cc:1467] Processing CreateTablet for tablet e2b27de4d2d84a2f8c177a231c985785 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.479228 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b27de4d2d84a2f8c177a231c985785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.479712 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 (DEFAULT_TABLE table=default.Table_375f890255ed48659c41d4c11bb30578 [id=8029d1e576b048a39b876990392bafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.480409 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323b96ae7d1e471998a39f3be0857ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.482055 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 43225934cef04109b3dbc057b230f032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.484298 5686 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4993c287a40cf964877b5464372a6 (DEFAULT_TABLE table=default.Table_ced552bcfdd14427a0b6990736d3165c [id=54c1b8d97fd343f7b49e7aba123d9d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.484705 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4993c287a40cf964877b5464372a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.486483 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 (DEFAULT_TABLE table=default.Table_5ee780b3d01a481b80949a3ade7e3cd3 [id=dd2fd8b019214415861bddf4401af755]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.486984 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 924d8d2fe2f148cba13f107e4c70fc68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.490159 5700 tablet_service.cc:1467] Processing CreateTablet for tablet d8a297a72b474e508184e1f32a2a37cf (DEFAULT_TABLE table=default.Table_e61891ba7bda49f99a67145edfe93e5b [id=d5f1bf3c02354c16818b755027c0944c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.490588 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a297a72b474e508184e1f32a2a37cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.493824 5838 tablet_replica.cc:330] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:01.500483 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb (DEFAULT_TABLE table=default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.500831 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120bce4b20714fc1a0e1237abc00d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.502234 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.502918 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.503532 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.503877 5838 raft_consensus.cc:2229] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.504133 5838 raft_consensus.cc:2258] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.505048 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:01.505544 5838 ts_tablet_manager.cc:1875] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.507438 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e33e40ef44d06a359b82d8fe48157" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:01.507721 5718 raft_consensus.cc:2454] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:12:01.510385 5654 leader_election.cc:341] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.511034 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet e182d186e6fe4e388a68c61ccf159f8d with delete_type TABLET_DATA_DELETED (replaced by tablet 2b14c80b96bb43088e1d1354a1dbdd31) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.511945 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 3618b416194f4effb29b823709593359 (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.512665 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3618b416194f4effb29b823709593359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.512672 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9324570984372af7b9574f0cf455a (DEFAULT_TABLE table=default.Table_b01d5fc4ce904601b66e4e880c65d768 [id=f9c03e081d704f0b9f0c60b97a2265c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.513227 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 (DEFAULT_TABLE table=default.Table_375f890255ed48659c41d4c11bb30578 [id=8029d1e576b048a39b876990392bafca]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.515313 5561 tablet_service.cc:1467] Processing CreateTablet for tablet e2b27de4d2d84a2f8c177a231c985785 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.516396 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 89f80e643c59420a90d2d57ca0b703c7 (DEFAULT_TABLE table=default.Table_96bb14500b6740f7bc08506e19f12519 [id=19fa9ceb2fa1458cbbf10c878ffa1853]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.518889 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 323b96ae7d1e471998a39f3be0857ce7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.524147 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb (DEFAULT_TABLE table=default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.524681 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120bce4b20714fc1a0e1237abc00d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.525851 5698 tablet_service.cc:1467] Processing CreateTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa (DEFAULT_TABLE table=default.Table_616cd80c5f274e8ab4b39e0d82adf728 [id=b54202d26e9846f9b557bf020ee1e807]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.526290 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9a65a8e939f4b7d89d00720845ea1fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.527315 5690 tablet_service.cc:1467] Processing CreateTablet for tablet b1d5627f71694fb68a15fb186c9068ce (DEFAULT_TABLE table=default.Table_63548a17f66844648a2d71fb0c7d7175 [id=a3a382d2cd854dba89a7be0230de2522]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.528079 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1d5627f71694fb68a15fb186c9068ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.530972 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 07a9324570984372af7b9574f0cf455a (table default.Table_a2c072fbeb35480fa276c46cbc1c6aa7 [id=f9c03e081d704f0b9f0c60b97a2265c0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 07a9324570984372af7b9574f0cf455a consensus state has no leader
W20240507 08:12:01.532528 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 43225934cef04109b3dbc057b230f032 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43225934cef04109b3dbc057b230f032 already in progress: creating tablet
W20240507 08:12:01.533386 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 957775c01be94cc081db344c95a81310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 957775c01be94cc081db344c95a81310 already in progress: creating tablet
W20240507 08:12:01.533766 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b8935f7c20ed43a69e9cfcb28fe5d023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8935f7c20ed43a69e9cfcb28fe5d023 already in progress: creating tablet
W20240507 08:12:01.534083 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6563c42d240b4e8685dfa4972f2b9924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6563c42d240b4e8685dfa4972f2b9924 already in progress: creating tablet
I20240507 08:12:01.443396 5843 ts_tablet_manager.cc:1398] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:01.535741 5843 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:01.541433 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9324570984372af7b9574f0cf455a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.453032 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e2b27de4d2d84a2f8c177a231c985785 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.544791 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b27de4d2d84a2f8c177a231c985785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.547480 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:01.556366 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7818e3f7c1234bf6827826172d1e6af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.561007 6409 raft_consensus.cc:482] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.563299 6409 raft_consensus.cc:504] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.564348 6409 leader_election.cc:288] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:01.565021 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 54732.1us
I20240507 08:12:01.572922 5843 tablet_bootstrap.cc:654] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.594168 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet e182d186e6fe4e388a68c61ccf159f8d with delete_type TABLET_DATA_DELETED (replaced by tablet 2b14c80b96bb43088e1d1354a1dbdd31) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.596743 5841 raft_consensus.cc:2229] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.456643 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 89f80e643c59420a90d2d57ca0b703c7 (DEFAULT_TABLE table=default.Table_96bb14500b6740f7bc08506e19f12519 [id=19fa9ceb2fa1458cbbf10c878ffa1853]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:01.612005 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b8ebbcf0d5a54d63a96a9374e839af83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8ebbcf0d5a54d63a96a9374e839af83 already in progress: creating tablet
I20240507 08:12:01.615403 5565 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4993c287a40cf964877b5464372a6 (DEFAULT_TABLE table=default.Table_ced552bcfdd14427a0b6990736d3165c [id=54c1b8d97fd343f7b49e7aba123d9d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.621255 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.623600 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2b27de4d2d84a2f8c177a231c985785. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.627281 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.460711 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 07a9324570984372af7b9574f0cf455a (DEFAULT_TABLE table=default.Table_b01d5fc4ce904601b66e4e880c65d768 [id=f9c03e081d704f0b9f0c60b97a2265c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.463944 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb (DEFAULT_TABLE table=default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.468102 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 3618b416194f4effb29b823709593359 (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.491503 5422 tablet_service.cc:1467] Processing CreateTablet for tablet b5d4993c287a40cf964877b5464372a6 (DEFAULT_TABLE table=default.Table_ced552bcfdd14427a0b6990736d3165c [id=54c1b8d97fd343f7b49e7aba123d9d1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.630686 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 3618b416194f4effb29b823709593359 (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.631044 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.638494 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.638787 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.639312 5841 raft_consensus.cc:2258] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.639734 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f80e643c59420a90d2d57ca0b703c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.640110 5841 ts_tablet_manager.cc:1875] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.640955 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4993c287a40cf964877b5464372a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.641736 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9324570984372af7b9574f0cf455a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.642421 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.642416 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.642736 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 120bce4b20714fc1a0e1237abc00d5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.643424 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "916e33e40ef44d06a359b82d8fe48157" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:01.656425 6409 raft_consensus.cc:482] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.656618 6409 raft_consensus.cc:504] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.663996 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3618b416194f4effb29b823709593359. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.665812 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet e182d186e6fe4e388a68c61ccf159f8d with delete_type TABLET_DATA_DELETED (replaced by tablet 2b14c80b96bb43088e1d1354a1dbdd31) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.666115 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 6563c42d240b4e8685dfa4972f2b9924 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.509724 5377 leader_election.cc:302] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:01.667672 5377 raft_consensus.cc:2670] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:01.667768 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 158080us
W20240507 08:12:01.669647 5378 leader_election.cc:341] T 916e33e40ef44d06a359b82d8fe48157 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.673635 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89f80e643c59420a90d2d57ca0b703c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.685819 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b8935f7c20ed43a69e9cfcb28fe5d023 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.687806 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd02a49359743c4b4e540ab0338d8ed" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:01.689237 5546 tablet_service.cc:1467] Processing CreateTablet for tablet d8a297a72b474e508184e1f32a2a37cf (DEFAULT_TABLE table=default.Table_e61891ba7bda49f99a67145edfe93e5b [id=d5f1bf3c02354c16818b755027c0944c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.691704 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:01.501662 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:01.699352 5843 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:01.703584 5839 tablet_bootstrap.cc:492] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.703801 5839 ts_tablet_manager.cc:1367] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.552s user 0.007s sys 0.000s
I20240507 08:12:01.705564 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5d4993c287a40cf964877b5464372a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.706328 5839 raft_consensus.cc:348] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.706646 5839 raft_consensus.cc:374] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.707327 5839 raft_consensus.cc:729] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.707707 5839 consensus_queue.cc:260] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.499078 5426 tablet_service.cc:1467] Processing CreateTablet for tablet d8a297a72b474e508184e1f32a2a37cf (DEFAULT_TABLE table=default.Table_e61891ba7bda49f99a67145edfe93e5b [id=d5f1bf3c02354c16818b755027c0944c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.714756 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07a9324570984372af7b9574f0cf455a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.733692 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3618b416194f4effb29b823709593359. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:01.740628 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e182d186e6fe4e388a68c61ccf159f8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e182d186e6fe4e388a68c61ccf159f8d already in progress: creating tablet
I20240507 08:12:01.742162 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a13969f58dd2425f9f93a2fadbdd178a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.742712 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a13969f58dd2425f9f93a2fadbdd178a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.743570 5843 ts_tablet_manager.cc:1367] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.008s
I20240507 08:12:01.747516 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e3df44d5188247b6a317ba4185f0b745" } modify_external_catalogs: true
I20240507 08:12:01.747771 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e3df44d5188247b6a317ba4185f0b745" } modify_external_catalogs: true
I20240507 08:12:01.748735 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_04756d6d06eb47d2a3c61debf3d6eef1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.749267 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04756d6d06eb47d2a3c61debf3d6eef1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.749984 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd02a49359743c4b4e540ab0338d8ed" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:01.751725 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8c28748ca00b43e7bff97d436034daaf" } new_table_name: "default.Table_2d4d839694e44fbdba0344b3d11b5555" modify_external_catalogs: true
I20240507 08:12:01.756276 6376 raft_consensus.cc:482] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.756477 6376 raft_consensus.cc:504] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.757036 6376 leader_election.cc:288] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.757813 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612c023740644cb9564c44f7bfe8b61" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.758633 5654 leader_election.cc:341] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.760522 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a297a72b474e508184e1f32a2a37cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.763674 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 53fb38b567c5455ebc441e5f71db694f from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:01.764281 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b5e3d23bbb6439a87b303dff1b44a1d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:01.768646 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8a297a72b474e508184e1f32a2a37cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.770648 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4f4c310282004fbe930f8b2aa57ed2c0" } modify_external_catalogs: true
I20240507 08:12:01.770855 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4f4c310282004fbe930f8b2aa57ed2c0" } modify_external_catalogs: true
I20240507 08:12:01.771840 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b21b58e8d49943268e7491117d5d717e" } modify_external_catalogs: true
I20240507 08:12:01.772004 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b21b58e8d49943268e7491117d5d717e" } modify_external_catalogs: true
I20240507 08:12:01.781770 6409 leader_election.cc:288] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:01.782850 5839 ts_tablet_manager.cc:1398] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.079s user 0.002s sys 0.001s
I20240507 08:12:01.783282 5839 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:01.785343 5839 tablet_bootstrap.cc:654] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.786075 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a3e97a9c990549b880b71e973f807616" } modify_external_catalogs: true
I20240507 08:12:01.786316 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a3e97a9c990549b880b71e973f807616" } modify_external_catalogs: true
W20240507 08:12:01.792804 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8e6454c4a59d4c4cae1681bcae0da2d7 consensus state has no leader
I20240507 08:12:01.795527 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eec3f4ed076b4741aa5573b9805a83ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.796082 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eec3f4ed076b4741aa5573b9805a83ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:01.801332 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2013559307a34443ac998dab956c55f9 consensus state has no leader
I20240507 08:12:01.804225 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fdbd31dead3043cba3f6dce0b222bd13" } new_table_name: "default.Table_02ac12aae7304546b72a2783e522a876" modify_external_catalogs: true
W20240507 08:12:01.812088 5248 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 71152.2us
I20240507 08:12:01.815553 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_be493cfc9faf43faa5b0a4afe7287d6d" } new_table_name: "default.Table_2b0061a7e41c4d66aaff6de6bad0e892" modify_external_catalogs: true
I20240507 08:12:01.825947 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9a8c1adf5d32474c9d849f39a54d8e84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.826568 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a8c1adf5d32474c9d849f39a54d8e84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.827852 6409 raft_consensus.cc:482] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.831055 5843 raft_consensus.cc:348] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:01.835371 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4e37b73134cf42ad84b6eec90c9b805f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e37b73134cf42ad84b6eec90c9b805f already in progress: creating tablet
I20240507 08:12:01.834712 6376 raft_consensus.cc:482] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:01.839519 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0542325c80241fca7e317cda64302a9 consensus state has no leader
I20240507 08:12:01.843564 6409 raft_consensus.cc:504] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.844228 6409 leader_election.cc:288] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:01.844671 5838 ts_tablet_manager.cc:1888] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.844895 5838 log.cc:1196] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/8e3ffaf1781e4b108f305346476905be
I20240507 08:12:01.845495 5838 ts_tablet_manager.cc:1909] T 8e3ffaf1781e4b108f305346476905be P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:01.846215 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 8e3ffaf1781e4b108f305346476905be (table default.Table_5aa2ec19328a431f8277539512fac60e [id=4f2157f8351143c29c6db4dd3843a67d]) successfully deleted
I20240507 08:12:01.847177 5843 raft_consensus.cc:374] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.847282 5843 raft_consensus.cc:729] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.847486 5843 consensus_queue.cc:260] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.847996 5843 ts_tablet_manager.cc:1398] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.103s user 0.002s sys 0.000s
I20240507 08:12:01.848321 5843 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:01.857988 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612c023740644cb9564c44f7bfe8b61" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:01.859440 5843 tablet_bootstrap.cc:654] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.876094 5841 ts_tablet_manager.cc:1888] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:01.876252 5841 log.cc:1196] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:01.876717 5841 ts_tablet_manager.cc:1909] T d537bed29a2f47058255d2530b5c0d23 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:01.877267 5839 tablet_bootstrap.cc:492] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.877444 5839 ts_tablet_manager.cc:1367] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.094s user 0.000s sys 0.005s
I20240507 08:12:01.877739 5841 tablet_replica.cc:330] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:01.878024 5841 raft_consensus.cc:2229] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:01.878248 5841 raft_consensus.cc:2258] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:01.878361 5839 raft_consensus.cc:348] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.878612 5839 raft_consensus.cc:374] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.878685 5839 raft_consensus.cc:729] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.879004 5841 ts_tablet_manager.cc:1875] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:01.878928 5839 consensus_queue.cc:260] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.879441 5839 ts_tablet_manager.cc:1398] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240507 08:12:01.884883 5642 kernel_stack_watchdog.cc:198] Thread 5840 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:637 for 429ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f2d52363980 (unknown) at ??:0
@ 0x90ef77 __sanitizer::internal_sched_yield() at ??:?
@ 0x87462d __interceptor_pthread_create at ??:?
@ 0x7f2d4efd1d11 kudu::Thread::StartThread() at ??:0
@ 0xa4d187 kudu::Thread::Create() at ??:?
@ 0x7f2d4eff27ab kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f2d4efee3ad kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f2d4eff34ff kudu::ThreadPool::Submit() at ??:0
@ 0x7f2d5a1984ad kudu::log::SegmentAllocator::AsyncAllocateSegmentUnlocked() at ??:0
@ 0x7f2d5a19749f kudu::log::SegmentAllocator::AllocateSegmentAndRollOver() at ??:0
@ 0x7f2d5a197070 kudu::log::SegmentAllocator::Init() at ??:0
@ 0x7f2d5a19ef85 kudu::log::Log::Init() at ??:0
@ 0x7f2d5a19e2e1 kudu::log::Log::Open() at ??:0
@ 0x7f2d5ba95fd1 kudu::tablet::TabletBootstrap::OpenNewLog() at ??:0
@ 0x7f2d5ba9233c kudu::tablet::TabletBootstrap::RunBootstrap() at ??:0
@ 0x7f2d5ba90abf kudu::tablet::TabletBootstrap::Bootstrap() at ??:0
I20240507 08:12:01.882858 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_36a39b6cb68c403d8d67600517f6bb5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:01.885941 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36a39b6cb68c403d8d67600517f6bb5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:01.889240 5839 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:01.885166 6376 raft_consensus.cc:504] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.890604 6376 leader_election.cc:288] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.890956 5839 tablet_bootstrap.cc:654] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:01.891923 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 43225934cef04109b3dbc057b230f032 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43225934cef04109b3dbc057b230f032 already in progress: creating tablet
I20240507 08:12:01.892254 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee3a6c4ac642a3921560ea13bc1341" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.895434 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4e37b73134cf42ad84b6eec90c9b805f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e37b73134cf42ad84b6eec90c9b805f already in progress: creating tablet
W20240507 08:12:01.896886 5654 leader_election.cc:341] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:01.898710 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e182d186e6fe4e388a68c61ccf159f8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e182d186e6fe4e388a68c61ccf159f8d already in progress: creating tablet
W20240507 08:12:01.901732 5840 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.651s user 0.148s sys 0.191s
W20240507 08:12:01.901841 5840 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.651s user 0.148s sys 0.191s
I20240507 08:12:01.903272 5840 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.903426 5840 ts_tablet_manager.cc:1367] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.657s user 0.149s sys 0.191s
I20240507 08:12:01.904331 5840 raft_consensus.cc:348] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.904536 5840 raft_consensus.cc:374] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.904639 5840 raft_consensus.cc:729] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:01.904685 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6563c42d240b4e8685dfa4972f2b9924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6563c42d240b4e8685dfa4972f2b9924 already in progress: creating tablet
I20240507 08:12:01.904870 5840 consensus_queue.cc:260] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.906129 5839 tablet_bootstrap.cc:492] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.906261 5839 ts_tablet_manager.cc:1367] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
W20240507 08:12:01.908265 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b8935f7c20ed43a69e9cfcb28fe5d023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8935f7c20ed43a69e9cfcb28fe5d023 already in progress: creating tablet
I20240507 08:12:01.908311 5839 raft_consensus.cc:348] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.908507 5839 raft_consensus.cc:374] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.908609 5839 raft_consensus.cc:729] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.908824 5839 consensus_queue.cc:260] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:01.913292 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 957775c01be94cc081db344c95a81310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 957775c01be94cc081db344c95a81310 already in progress: creating tablet
I20240507 08:12:01.915060 5839 ts_tablet_manager.cc:1398] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:01.915375 5839 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:01.916914 5839 tablet_bootstrap.cc:654] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.918255 5840 ts_tablet_manager.cc:1398] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240507 08:12:01.918593 5840 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.920720 5840 tablet_bootstrap.cc:654] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.925948 5839 tablet_bootstrap.cc:492] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.926105 5839 ts_tablet_manager.cc:1367] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240507 08:12:01.926982 5839 raft_consensus.cc:348] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.927189 5839 raft_consensus.cc:374] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.927266 5839 raft_consensus.cc:729] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.927505 5839 consensus_queue.cc:260] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.930042 5260 catalog_manager.cc:5484] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:01.930601 5260 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet b2de528d9fc046f183f70388cffab6c8 (table default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a]). Expected version 2 got 0
I20240507 08:12:01.931257 5840 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.931979 5840 ts_tablet_manager.cc:1367] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.002s
I20240507 08:12:01.932736 5840 raft_consensus.cc:348] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.932960 5840 raft_consensus.cc:374] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.933032 5840 raft_consensus.cc:729] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.933271 5840 consensus_queue.cc:260] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.933954 5840 ts_tablet_manager.cc:1398] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:01.934262 5840 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.935873 5840 tablet_bootstrap.cc:654] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.939815 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad95383487e458ca162038f6940271b (DEFAULT_TABLE table=default.Table_20e0ec532f1549dd9c1df7c636346293 [id=5ef4d1534973481182353514d1a762be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.940230 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad95383487e458ca162038f6940271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.940363 6496 raft_consensus.cc:482] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:01.940562 6496 raft_consensus.cc:504] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:01.941556 6496 leader_election.cc:288] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:01.941905 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4 (DEFAULT_TABLE table=default.Table_109dc998f0b24d49824e20c1693703b7 [id=6b441720033242f0a8cbbe63081fb128]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.942327 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.942322 5839 ts_tablet_manager.cc:1398] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240507 08:12:01.942615 5839 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:01.944586 5839 tablet_bootstrap.cc:654] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:01.945852 5692 tablet_service.cc:1467] Processing CreateTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 (DEFAULT_TABLE table=default.Table_d23b6a487a6a4a20876771868a0d439a [id=7fff7579bf9141c7a0e6770aba33b1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.946179 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b60e7fd01c4c96911fff6bf62ade96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.949872 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6aeb8da56047adbccda3cfbe00e557 (DEFAULT_TABLE table=default.Table_e7d9305c31714aba91c87411c2855c6f [id=a265951d6865448fb2bc6a5b345a30e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.950294 6481 raft_consensus.cc:482] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:01.950373 5654 consensus_peers.cc:485] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:01.950325 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6aeb8da56047adbccda3cfbe00e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.950508 6481 raft_consensus.cc:504] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:01.951462 5517 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 6f6670e99e4a4442801d42950eb7ab79
I20240507 08:12:01.951529 6481 leader_election.cc:288] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:01.951928 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:01.952450 5654 leader_election.cc:341] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:01.960752 5840 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:01.963516 5840 ts_tablet_manager.cc:1367] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20240507 08:12:01.964476 5840 raft_consensus.cc:348] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.964663 5840 raft_consensus.cc:374] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:01.964746 5840 raft_consensus.cc:729] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:01.964879 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 (DEFAULT_TABLE table=default.Table_0f91e3e5674d49efa25f246c35d4f912 [id=e522451cc4284d3086eb9a627beac89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.965287 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8497f8ea3c9a48d281d91c90ea305cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.966015 5692 tablet_service.cc:1467] Processing CreateTablet for tablet ab9f50b062ba42e99c471f9a70024c55 (DEFAULT_TABLE table=default.Table_91931ba97b4c48ae92cb1ecb56850bd7 [id=1e9ebbcfba02494394f6884a883f8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.966452 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9f50b062ba42e99c471f9a70024c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.968218 5840 consensus_queue.cc:260] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.968875 5840 ts_tablet_manager.cc:1398] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:01.969147 5840 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:01.914319 5444 raft_consensus.cc:2454] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:01.970582 5839 tablet_bootstrap.cc:492] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:01.970722 5839 ts_tablet_manager.cc:1367] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20240507 08:12:01.973799 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 0b47825cbd714b91b729118e5f01eb16 (DEFAULT_TABLE table=default.Table_ff9a2c687f9446dab1f59c2df5d949bf [id=a1b363da2588402f874cee52d9b7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.974273 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 128c2034d917473eb5c496319fdc9625 (DEFAULT_TABLE table=default.Table_e860ab852fe14a5993248381d5e7e17d [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.974404 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad95383487e458ca162038f6940271b (DEFAULT_TABLE table=default.Table_20e0ec532f1549dd9c1df7c636346293 [id=5ef4d1534973481182353514d1a762be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.974618 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9cecc600964ee1b77098abce0e0a51 (DEFAULT_TABLE table=default.Table_8083493fc8df43ee8867ad0dca71c295 [id=b0a24efbaecb4ca99a6b7c34183adc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.974946 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 690b258a9b564ed1b07aef29a774b749 (DEFAULT_TABLE table=default.Table_cf9647a579774e5ebc3de8cc3754fbef [id=e34309ce25b54ce89c5554b352fb42e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.975312 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4 (DEFAULT_TABLE table=default.Table_109dc998f0b24d49824e20c1693703b7 [id=6b441720033242f0a8cbbe63081fb128]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.975554 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b47825cbd714b91b729118e5f01eb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.976188 5551 tablet_service.cc:1467] Processing CreateTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 (DEFAULT_TABLE table=default.Table_d23b6a487a6a4a20876771868a0d439a [id=7fff7579bf9141c7a0e6770aba33b1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.976656 5698 tablet_service.cc:1467] Processing CreateTablet for tablet a7fb0c0ec64847cb8552ebabc09e2f5d (DEFAULT_TABLE table=default.Table_fa1ba71ce9b5486f98e731e1ede13b36 [id=fe347e2503a44bb1bb51bb234d792089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.977001 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6aeb8da56047adbccda3cfbe00e557 (DEFAULT_TABLE table=default.Table_e7d9305c31714aba91c87411c2855c6f [id=a265951d6865448fb2bc6a5b345a30e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.977036 5688 tablet_service.cc:1467] Processing CreateTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 (DEFAULT_TABLE table=default.Table_48a8b33cbac2439689197979206121b0 [id=ab86eb1ef96c4ac0aa657fa10db25d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.977319 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c2034d917473eb5c496319fdc9625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.977353 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6aeb8da56047adbccda3cfbe00e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.977694 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fb0c0ec64847cb8552ebabc09e2f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.978554 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9cecc600964ee1b77098abce0e0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.975471 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee3a6c4ac642a3921560ea13bc1341" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:01.978844 5443 raft_consensus.cc:2454] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:01.979722 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690b258a9b564ed1b07aef29a774b749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.980206 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b60e7fd01c4c96911fff6bf62ade96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.980417 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bb96bb9d8f4930b531a3dd30d733c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.980614 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:01.980810 5443 raft_consensus.cc:2454] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:01.981796 5656 leader_election.cc:302] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.982239 5656 leader_election.cc:302] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.982507 6376 raft_consensus.cc:2781] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:01.982643 5656 leader_election.cc:302] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:01.982672 6376 raft_consensus.cc:482] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:01.982806 6376 raft_consensus.cc:3037] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.982996 6481 raft_consensus.cc:2781] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:01.983111 6481 raft_consensus.cc:482] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:01.983202 6481 raft_consensus.cc:3037] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.985841 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d38bf47908640beb3f6d94ad4446717" } modify_external_catalogs: true
I20240507 08:12:01.986033 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d38bf47908640beb3f6d94ad4446717" } modify_external_catalogs: true
I20240507 08:12:01.986186 6376 raft_consensus.cc:504] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:01.987279 6376 leader_election.cc:288] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:01.987480 6376 raft_consensus.cc:2781] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240507 08:12:01.987505 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04d5866ebb974ee6842d8c66671a8467 consensus state has no leader
I20240507 08:12:01.987562 6376 raft_consensus.cc:482] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:01.987711 6376 raft_consensus.cc:3037] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:01.987747 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:01.993809 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 4ad95383487e458ca162038f6940271b (DEFAULT_TABLE table=default.Table_20e0ec532f1549dd9c1df7c636346293 [id=5ef4d1534973481182353514d1a762be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.994156 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad95383487e458ca162038f6940271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.997619 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4 (DEFAULT_TABLE table=default.Table_109dc998f0b24d49824e20c1693703b7 [id=6b441720033242f0a8cbbe63081fb128]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.998138 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.998169 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 (DEFAULT_TABLE table=default.Table_0f91e3e5674d49efa25f246c35d4f912 [id=e522451cc4284d3086eb9a627beac89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.998467 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8497f8ea3c9a48d281d91c90ea305cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:01.998724 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 1e6aeb8da56047adbccda3cfbe00e557 (DEFAULT_TABLE table=default.Table_e7d9305c31714aba91c87411c2855c6f [id=a265951d6865448fb2bc6a5b345a30e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:01.999090 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e6aeb8da56047adbccda3cfbe00e557. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.000082 6481 raft_consensus.cc:504] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.000391 5411 tablet_service.cc:1467] Processing CreateTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 (DEFAULT_TABLE table=default.Table_48a8b33cbac2439689197979206121b0 [id=ab86eb1ef96c4ac0aa657fa10db25d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.000712 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bb96bb9d8f4930b531a3dd30d733c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.000787 6481 leader_election.cc:288] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:02.000768 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9cecc600964ee1b77098abce0e0a51 (DEFAULT_TABLE table=default.Table_8083493fc8df43ee8867ad0dca71c295 [id=b0a24efbaecb4ca99a6b7c34183adc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.001083 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9cecc600964ee1b77098abce0e0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.001798 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 128c2034d917473eb5c496319fdc9625 (DEFAULT_TABLE table=default.Table_e860ab852fe14a5993248381d5e7e17d [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.002090 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c2034d917473eb5c496319fdc9625. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.005795 6376 raft_consensus.cc:504] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:02.006366 6376 leader_election.cc:288] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:02.006873 5423 tablet_service.cc:1467] Processing CreateTablet for tablet a7fb0c0ec64847cb8552ebabc09e2f5d (DEFAULT_TABLE table=default.Table_fa1ba71ce9b5486f98e731e1ede13b36 [id=fe347e2503a44bb1bb51bb234d792089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.007162 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fb0c0ec64847cb8552ebabc09e2f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.010021 5414 tablet_service.cc:1467] Processing CreateTablet for tablet ab9f50b062ba42e99c471f9a70024c55 (DEFAULT_TABLE table=default.Table_91931ba97b4c48ae92cb1ecb56850bd7 [id=1e9ebbcfba02494394f6884a883f8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.010221 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.010327 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9f50b062ba42e99c471f9a70024c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.011338 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d537bed29a2f47058255d2530b5c0d23 (table default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]) successfully deleted
I20240507 08:12:02.011588 5425 tablet_service.cc:1467] Processing CreateTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 (DEFAULT_TABLE table=default.Table_d23b6a487a6a4a20876771868a0d439a [id=7fff7579bf9141c7a0e6770aba33b1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.011858 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b60e7fd01c4c96911fff6bf62ade96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.013743 6499 raft_consensus.cc:482] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.013873 6499 raft_consensus.cc:504] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.014524 6499 leader_election.cc:288] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:02.015681 6499 raft_consensus.cc:482] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.015796 6499 raft_consensus.cc:504] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.016368 6499 leader_election.cc:288] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:02.018378 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 690b258a9b564ed1b07aef29a774b749 (DEFAULT_TABLE table=default.Table_cf9647a579774e5ebc3de8cc3754fbef [id=e34309ce25b54ce89c5554b352fb42e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.022022 5693 tablet_service.cc:1467] Processing CreateTablet for tablet c7d19b7cc39a4337826819a20205aecc (DEFAULT_TABLE table=default.Table_682aa156946f429890d5095fa23f097e [id=e49233a513cb4d86bc2a503ceec7c81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.023782 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690b258a9b564ed1b07aef29a774b749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.028100 5426 tablet_service.cc:1467] Processing CreateTablet for tablet c7d19b7cc39a4337826819a20205aecc (DEFAULT_TABLE table=default.Table_682aa156946f429890d5095fa23f097e [id=e49233a513cb4d86bc2a503ceec7c81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.028697 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d19b7cc39a4337826819a20205aecc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.029484 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85dc58133d2b4f5496b4e570ae6494c0 consensus state has no leader
I20240507 08:12:02.029560 6498 raft_consensus.cc:482] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.030982 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 14929efb0ed049d6b554598a5b8cb494
I20240507 08:12:02.034350 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e298f11ff9f54e48b8de435f6849f39a
I20240507 08:12:02.034757 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 234e582b211940aea3471e5accb9d016
I20240507 08:12:02.035480 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 62f3b6d22f7840e7b43da7a52c25a810
I20240507 08:12:02.038240 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:02.043152 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e582b211940aea3471e5accb9d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.043428 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 0b47825cbd714b91b729118e5f01eb16 (DEFAULT_TABLE table=default.Table_ff9a2c687f9446dab1f59c2df5d949bf [id=a1b363da2588402f874cee52d9b7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.043520 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.043732 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.043910 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b47825cbd714b91b729118e5f01eb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.046845 6498 raft_consensus.cc:504] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:02.031706 5377 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:02.049124 6499 consensus_queue.cc:1035] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:02.051098 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.051779 5843 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:02.051895 5843 ts_tablet_manager.cc:1367] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.015s
I20240507 08:12:02.052006 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.052695 6498 leader_election.cc:288] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:02.052587 5843 raft_consensus.cc:348] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.052774 5843 raft_consensus.cc:374] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:02.052819 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.052834 5843 raft_consensus.cc:729] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.053017 5843 consensus_queue.cc:260] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.053478 5843 ts_tablet_manager.cc:1398] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:02.053920 5843 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:02.062315 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d19b7cc39a4337826819a20205aecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.065093 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:02.068936 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193ec0b463d04835b74df80229c85cde" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:02.071395 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:02.073138 5717 raft_consensus.cc:2379] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc665657d5b34fd7aec884e0f2668e11 in current term 1: Already voted for candidate 1cbb887175f345ada2433adaec906fd4 in this term.
I20240507 08:12:02.073310 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22334378eec4ab3885d0f9d2d812a3e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:02.074005 5840 tablet_bootstrap.cc:654] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:02.077046 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:02.081341 5720 raft_consensus.cc:2454] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:02.085706 5843 tablet_bootstrap.cc:654] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:02.086690 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e582b211940aea3471e5accb9d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:02.087260 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:02.087731 5377 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:02.088883 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:02.089159 5377 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:02.089408 5377 leader_election.cc:341] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:12:02.089545 6498 consensus_queue.cc:1035] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:02.091523 5377 leader_election.cc:302] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:02.091590 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:02.091773 6499 raft_consensus.cc:2781] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:02.091812 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:02.091836 6499 raft_consensus.cc:482] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:02.091897 6499 raft_consensus.cc:3037] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:02.092005 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 (DEFAULT_TABLE table=default.Table_f8765d860074430f9fb9ecfba92d9dd7 [id=bf6530b2bb8544a48f653814f9855a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.092854 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f75cf9f00b4d03b72101f1a0668df3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.132329 5083 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskrbFV97/build/asan/bin/kudu pid 5940: Timed out: CheckLeaks RPC to 127.4.246.253:43459 timed out after 1.000s (SENT)
I20240507 08:12:02.507306 5083 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskrbFV97/build/asan/bin/kudu with pid 5940
W20240507 08:12:02.284958 6301 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:34376 (request call id 0) took 815 ms (client timeout 1000 ms). Trace:
I20240507 08:12:02.111775 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fe8a85dddcfb41a18221367878c0dac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:02.096429 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:02.095615 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d3daa5cf7d5428eb3baf74b7c108eb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.510015 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ad95383487e458ca162038f6940271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.094164 6499 raft_consensus.cc:504] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.518532 5543 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 119) took 1052 ms. Trace:
I20240507 08:12:02.112208 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1f68c8dc6aa54480bac7fcc19cd692e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:02.518668 5543 rpcz_store.cc:276] 0507 08:12:01.466332 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.466698 (+ 366us) service_pool.cc:225] Handling call
0507 08:12:01.841510 (+374812us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x61800015508c. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2c41a12e 00007f2f2c22079f 00007f2f2ec1e768 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58
0507 08:12:02.518483 (+676973us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":61754752}
W20240507 08:12:02.519279 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:02.141788 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 159136us
I20240507 08:12:02.096136 5839 raft_consensus.cc:348] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.149327 5840 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:02.521463 5839 raft_consensus.cc:374] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:02.097934 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612c023740644cb9564c44f7bfe8b61" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:02.098378 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee3a6c4ac642a3921560ea13bc1341" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:02.521685 5840 ts_tablet_manager.cc:1367] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.553s user 0.007s sys 0.006s
I20240507 08:12:02.098640 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 (DEFAULT_TABLE table=default.Table_0f91e3e5674d49efa25f246c35d4f912 [id=e522451cc4284d3086eb9a627beac89d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.114986 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bbc152c3898a4dbcbaa844b31b5598cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:02.098791 5565 tablet_service.cc:1467] Processing CreateTablet for tablet ab9f50b062ba42e99c471f9a70024c55 (DEFAULT_TABLE table=default.Table_91931ba97b4c48ae92cb1ecb56850bd7 [id=1e9ebbcfba02494394f6884a883f8b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.522581 5840 raft_consensus.cc:348] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:02.522703 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbc152c3898a4dbcbaa844b31b5598cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:02.522740 5654 leader_election.cc:341] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:02.522812 5840 raft_consensus.cc:374] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:02.100740 5561 tablet_service.cc:1467] Processing CreateTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 (DEFAULT_TABLE table=default.Table_48a8b33cbac2439689197979206121b0 [id=ab86eb1ef96c4ac0aa657fa10db25d68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.522883 5840 raft_consensus.cc:729] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.101123 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:02.101485 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 128c2034d917473eb5c496319fdc9625 (DEFAULT_TABLE table=default.Table_e860ab852fe14a5993248381d5e7e17d [id=af8257bea7d3403db47906d0fe2c79df]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.523116 5840 consensus_queue.cc:260] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.105499 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 690b258a9b564ed1b07aef29a774b749 (DEFAULT_TABLE table=default.Table_cf9647a579774e5ebc3de8cc3754fbef [id=e34309ce25b54ce89c5554b352fb42e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.189075 6513 tablet_replica.cc:330] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:02.105922 5562 tablet_service.cc:1467] Processing CreateTablet for tablet c7d19b7cc39a4337826819a20205aecc (DEFAULT_TABLE table=default.Table_682aa156946f429890d5095fa23f097e [id=e49233a513cb4d86bc2a503ceec7c81a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.106052 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 0b47825cbd714b91b729118e5f01eb16 (DEFAULT_TABLE table=default.Table_ff9a2c687f9446dab1f59c2df5d949bf [id=a1b363da2588402f874cee52d9b7749b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.106189 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 1b9cecc600964ee1b77098abce0e0a51 (DEFAULT_TABLE table=default.Table_8083493fc8df43ee8867ad0dca71c295 [id=b0a24efbaecb4ca99a6b7c34183adc7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.221105 6526 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:02.106354 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet 191adef34efa4c3999d4642a80177294 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:02.106527 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 (DEFAULT_TABLE table=default.Table_f8765d860074430f9fb9ecfba92d9dd7 [id=bf6530b2bb8544a48f653814f9855a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.106691 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 62f3b6d22f7840e7b43da7a52c25a810 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:02.106892 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:02.107018 5553 tablet_service.cc:1467] Processing CreateTablet for tablet a7fb0c0ec64847cb8552ebabc09e2f5d (DEFAULT_TABLE table=default.Table_fa1ba71ce9b5486f98e731e1ede13b36 [id=fe347e2503a44bb1bb51bb234d792089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.107189 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:02.108875 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:02.108999 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 234e582b211940aea3471e5accb9d016 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:02.524881 5654 leader_election.cc:341] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:02.116020 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0bba3367e45542dcb9de73f0f9a05a6f" } new_table_name: "default.Table_af114d78455749ce9330a056f6505e04" modify_external_catalogs: true
I20240507 08:12:02.112514 6496 raft_consensus.cc:482] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.116143 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1ba19248c0764ace86391198692f0bb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:02.528285 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ba19248c0764ace86391198692f0bb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.529129 6513 raft_consensus.cc:2229] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:02.095669 5520 leader_election.cc:341] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 6f6670e99e4a4442801d42950eb7ab79
I20240507 08:12:02.119920 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ab3f5d9e0ac34429a34b778d025e25f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:02.529347 6513 raft_consensus.cc:2258] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.288540 5551 tablet_service.cc:1467] Processing CreateTablet for tablet b340567fcccb48a495033c0c430b33b0 (DEFAULT_TABLE table=default.Table_36a39b6cb68c403d8d67600517f6bb5b [id=7b5b7fee15694eabb399705704ff3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.302099 5560 tablet_service.cc:1467] Processing CreateTablet for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 (DEFAULT_TABLE table=default.Table_9a8c1adf5d32474c9d849f39a54d8e84 [id=35d988e8eed14dad8783ccafffeb562f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:02.529659 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab3f5d9e0ac34429a34b778d025e25f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:02.393980 5506 kernel_stack_watchdog.cc:198] Thread 5520 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 177ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f2f22a83980 (unknown) at ??:0
@ 0x7f2f22a7d9c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f2f1ca03904 google::LogMessage::Flush() at ??:0
@ 0x7f2f1ca03a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f2f2aeaa5d3 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f2f2aead12a kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f2f2aeaceee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2f263438cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f2f263434ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f2f2632f4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f2f2632fa2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f2f262cc51a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f2f262caf45 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f2f262dd2c5 ev::base<>::method_thunk<>() at ??:0
I20240507 08:12:02.529953 6513 ts_tablet_manager.cc:1875] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.120543 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e667ff30b7ca4e44afc8675ec6981a0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:02.530929 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e667ff30b7ca4e44afc8675ec6981a0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.122291 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4929a2e8780d4124a8f206d62e5e5460"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:02.127606 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aa9ec40b8ef74e34994df20db4800454" } new_table_name: "default.Table_9328db0568cb47a2a28f09d770d78ab6" modify_external_catalogs: true
I20240507 08:12:02.138901 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6a9d7d23343543d9aa714b9fb7c0c006" } new_table_name: "default.Table_dea0e2b1b090454cbd15a00f7e792d6e" modify_external_catalogs: true
W20240507 08:12:02.186148 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 14929efb0ed049d6b554598a5b8cb494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14929efb0ed049d6b554598a5b8cb494 already in progress: creating tablet
W20240507 08:12:02.429776 5237 kernel_stack_watchdog.cc:198] Thread 5248 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 173ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f3bbc78f980 (unknown) at ??:0
@ 0x7f3bbc7899c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f3bb670f904 google::LogMessage::Flush() at ??:0
@ 0x7f3bb670fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f3bcc8b1465 kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7f3bcc808b09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f3bcc8b3e30 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f3bcc8b3c4e _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3bc004f8cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f3bc004f4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f3bc003b4e0 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f3bc003ba2c kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f3bbffd851a kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f3bbffd6f45 kudu::rpc::Connection::ReadHandler() at ??:0
I20240507 08:12:02.095873 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 (DEFAULT_TABLE table=default.Table_f8765d860074430f9fb9ecfba92d9dd7 [id=bf6530b2bb8544a48f653814f9855a5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.535359 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f75cf9f00b4d03b72101f1a0668df3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.535557 6499 leader_election.cc:288] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:02.533564 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4929a2e8780d4124a8f206d62e5e5460 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.125627 5841 ts_tablet_manager.cc:1888] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.539650 5841 log.cc:1196] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e1f9a56a95074e67bb33ceee1af48a18
W20240507 08:12:02.126839 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 464049us
I20240507 08:12:02.540000 5841 ts_tablet_manager.cc:1909] T e1f9a56a95074e67bb33ceee1af48a18 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:02.534708 5248 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 348571us
I20240507 08:12:02.540755 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 589) took 1360 ms. Trace:
I20240507 08:12:02.540858 5841 rpcz_store.cc:276] 0507 08:12:01.179942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.180625 (+ 683us) service_pool.cc:225] Handling call
0507 08:12:01.877499 (+696874us) ts_tablet_manager.cc:1120] Deleting tablet e1f9a56a95074e67bb33ceee1af48a18
0507 08:12:01.877506 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.047256 (+169750us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.106480 (+ 59224us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.540734 (+434254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":696676}
W20240507 08:12:02.541227 5378 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:02.541321 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 608) took 1295 ms. Trace:
I20240507 08:12:02.541396 5841 rpcz_store.cc:276] 0507 08:12:01.245753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.246251 (+ 498us) service_pool.cc:225] Handling call
0507 08:12:02.541210 (+1294959us) ts_tablet_manager.cc:1120] Deleting tablet 43225934cef04109b3dbc057b230f032
0507 08:12:02.541219 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.541308 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1294693}
I20240507 08:12:02.541745 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 621) took 1255 ms. Trace:
I20240507 08:12:02.541834 5841 rpcz_store.cc:276] 0507 08:12:01.286120 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.594080 (+307960us) service_pool.cc:225] Handling call
0507 08:12:02.541628 (+947548us) ts_tablet_manager.cc:1120] Deleting tablet e182d186e6fe4e388a68c61ccf159f8d
0507 08:12:02.541635 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.541731 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":947305}
I20240507 08:12:02.542129 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 623) took 1221 ms. Trace:
I20240507 08:12:02.542189 5841 rpcz_store.cc:276] 0507 08:12:01.320828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.627173 (+306345us) service_pool.cc:225] Handling call
0507 08:12:02.542035 (+914862us) ts_tablet_manager.cc:1120] Deleting tablet b8935f7c20ed43a69e9cfcb28fe5d023
0507 08:12:02.542042 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.542114 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":914508}
I20240507 08:12:02.542441 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 622) took 1221 ms. Trace:
I20240507 08:12:02.542506 5841 rpcz_store.cc:276] 0507 08:12:01.320703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.594340 (+273637us) service_pool.cc:225] Handling call
0507 08:12:02.542351 (+948011us) ts_tablet_manager.cc:1120] Deleting tablet 957775c01be94cc081db344c95a81310
0507 08:12:02.542360 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.542428 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":911157}
I20240507 08:12:02.542702 5841 tablet_replica.cc:330] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:02.542788 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 234e582b211940aea3471e5accb9d016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e582b211940aea3471e5accb9d016 already in progress: creating tablet
I20240507 08:12:02.542868 5841 raft_consensus.cc:2229] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:02.536113 5249 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (table default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8e6454c4a59d4c4cae1681bcae0da2d7 consensus state has no leader
I20240507 08:12:02.543026 5841 raft_consensus.cc:2258] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:02.543031 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 767905763ba14cb8bbc52151ae9f17e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 767905763ba14cb8bbc52151ae9f17e6 already in progress: creating tablet
W20240507 08:12:02.543221 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 62f3b6d22f7840e7b43da7a52c25a810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f3b6d22f7840e7b43da7a52c25a810 already in progress: creating tablet
W20240507 08:12:02.537425 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe8a85dddcfb41a18221367878c0dac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.543504 5841 ts_tablet_manager.cc:1875] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.541321 5378 leader_election.cc:302] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:02.545434 6499 raft_consensus.cc:2726] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:02.144065 5843 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:02.551435 5843 ts_tablet_manager.cc:1367] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.498s user 0.004s sys 0.016s
I20240507 08:12:02.551836 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f (DEFAULT_TABLE table=default.Table_8c20f0267d2b49e2a564ec31f3cd52a1 [id=2df70e8274d04362b0927028ab7456d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.552173 5843 raft_consensus.cc:348] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.552371 5843 raft_consensus.cc:374] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:02.552425 5843 raft_consensus.cc:729] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.552603 5843 consensus_queue.cc:260] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.553112 5843 ts_tablet_manager.cc:1398] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:02.553344 5843 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:02.221321 6525 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:02.296387 5426 tablet_service.cc:1467] Processing CreateTablet for tablet b340567fcccb48a495033c0c430b33b0 (DEFAULT_TABLE table=default.Table_36a39b6cb68c403d8d67600517f6bb5b [id=7b5b7fee15694eabb399705704ff3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.556056 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b340567fcccb48a495033c0c430b33b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.558825 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_96d7f5313ab8422bbfd790cea475d2ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:02.559374 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96d7f5313ab8422bbfd790cea475d2ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.307986 5425 tablet_service.cc:1467] Processing CreateTablet for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 (DEFAULT_TABLE table=default.Table_9a8c1adf5d32474c9d849f39a54d8e84 [id=35d988e8eed14dad8783ccafffeb562f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.323041 5414 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e (DEFAULT_TABLE table=default.Table_eec3f4ed076b4741aa5573b9805a83ce [id=8c0300f0ade143ad9d7681e4fe9c36fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.325268 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f (DEFAULT_TABLE table=default.Table_8c20f0267d2b49e2a564ec31f3cd52a1 [id=2df70e8274d04362b0927028ab7456d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.325268 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 124ede6467314270a2d99e0e45097c52 (DEFAULT_TABLE table=default.Table_04756d6d06eb47d2a3c61debf3d6eef1 [id=153882654fb24597a14188c78128e3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.325681 5427 tablet_service.cc:1467] Processing CreateTablet for tablet d2240fd14155495a8c97a0af96090faf (DEFAULT_TABLE table=default.Table_a13969f58dd2425f9f93a2fadbdd178a [id=69545cc630124f44878ef9314e167dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.326668 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 28e7cec189c7426795319154e4629356 (DEFAULT_TABLE table=default.Table_329152daa8064589a74946557194168f [id=23ca30ea7b534dacbb60dc2ad030ff6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.334650 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1f91718260466d99acfc9892dea536 (DEFAULT_TABLE table=default.Table_1462e4ac39014fb0aafe7485fe690e29 [id=e1821083c3ac493a8306413642160b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.335597 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 13fa1a807d3548d6836652379d455b0d (DEFAULT_TABLE table=default.Table_8f7c06393ac54a15842f97319969eff1 [id=7e854bf3351b4493a16b7bd7aa685baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.336663 5422 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5fc23bfb04c2db82d1549240334e6 (DEFAULT_TABLE table=default.Table_54232a23a7cc453290c1ee7b44906b9e [id=86ce23e55d1c4ba991c506dcdc2c9414]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.337509 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d (DEFAULT_TABLE table=default.Table_6cdc86ef760942ebbb3794342c9ccac7 [id=81b5c29c574a434ca6f2e6313052b635]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.337862 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 63685e9ef255477db589fa572b087b57 (DEFAULT_TABLE table=default.Table_2961308b2bc546de99dfee745feced1e [id=ab07a232d652404ab199d0ce8d42f527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.342916 5417 tablet_service.cc:1467] Processing CreateTablet for tablet bc029bc110b847f79d876ec3ab568832 (DEFAULT_TABLE table=default.Table_d67b71d4e235444db255a5638a555d43 [id=54d5d5e99cf94b4c8927453711d6e0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.343233 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.525676 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:02.566079 5446 raft_consensus.cc:3037] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:02.568619 5446 raft_consensus.cc:2454] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:02.568953 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e1f9a56a95074e67bb33ceee1af48a18 (table default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]) successfully deleted
W20240507 08:12:02.569221 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 43225934cef04109b3dbc057b230f032 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43225934cef04109b3dbc057b230f032 already in progress: creating tablet
W20240507 08:12:02.569470 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e182d186e6fe4e388a68c61ccf159f8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e182d186e6fe4e388a68c61ccf159f8d already in progress: creating tablet
W20240507 08:12:02.569690 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b8935f7c20ed43a69e9cfcb28fe5d023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8935f7c20ed43a69e9cfcb28fe5d023 already in progress: creating tablet
W20240507 08:12:02.569938 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 957775c01be94cc081db344c95a81310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 957775c01be94cc081db344c95a81310 already in progress: creating tablet
I20240507 08:12:02.527604 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2612c023740644cb9564c44f7bfe8b61" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:02.570168 5444 raft_consensus.cc:3037] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:02.572633 5444 raft_consensus.cc:2454] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:02.527801 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91ee3a6c4ac642a3921560ea13bc1341" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:02.572861 5443 raft_consensus.cc:3037] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:02.575170 5839 raft_consensus.cc:729] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.575800 5443 raft_consensus.cc:2454] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:02.576062 5843 tablet_bootstrap.cc:654] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:02.577863 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c56d06c6c4c9f83495859be8a5c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.578395 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:02.578647 5720 raft_consensus.cc:3037] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:02.580338 6496 raft_consensus.cc:504] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:02.580457 5520 leader_election.cc:302] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:02.581326 5564 tablet_service.cc:1467] Processing CreateTablet for tablet d2240fd14155495a8c97a0af96090faf (DEFAULT_TABLE table=default.Table_a13969f58dd2425f9f93a2fadbdd178a [id=69545cc630124f44878ef9314e167dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.581836 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:02.582463 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8497f8ea3c9a48d281d91c90ea305cb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.582906 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 28e7cec189c7426795319154e4629356 (DEFAULT_TABLE table=default.Table_329152daa8064589a74946557194168f [id=23ca30ea7b534dacbb60dc2ad030ff6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.583426 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e7cec189c7426795319154e4629356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.584388 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 124ede6467314270a2d99e0e45097c52 (DEFAULT_TABLE table=default.Table_04756d6d06eb47d2a3c61debf3d6eef1 [id=153882654fb24597a14188c78128e3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.585650 6508 consensus_queue.cc:1035] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:02.586649 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ede6467314270a2d99e0e45097c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.586979 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.587206 5549 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e (DEFAULT_TABLE table=default.Table_eec3f4ed076b4741aa5573b9805a83ce [id=8c0300f0ade143ad9d7681e4fe9c36fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.589795 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d7d8bcb635416693395b1f9ad27e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.593775 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ede6467314270a2d99e0e45097c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.597546 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2240fd14155495a8c97a0af96090faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.601792 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e7cec189c7426795319154e4629356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.605896 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1f91718260466d99acfc9892dea536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.609759 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fa1a807d3548d6836652379d455b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.609974 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63685e9ef255477db589fa572b087b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.613772 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc029bc110b847f79d876ec3ab568832. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.617766 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b14c80b96bb43088e1d1354a1dbdd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.584776 5839 consensus_queue.cc:260] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.621675 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482e4d0e762b488b8a0cc4e81c2c821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.625170 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193ec0b463d04835b74df80229c85cde" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:02.625509 6496 leader_election.cc:288] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:02.625717 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5fc23bfb04c2db82d1549240334e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.628463 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "907b5cb0e7a54903abe0b0d0e8d617dc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:02.625429 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22334378eec4ab3885d0f9d2d812a3e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:02.628878 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c56d06c6c4c9f83495859be8a5c5e. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.633962 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 37a23d112fcd4ea1b0c2f86133e82e32 consensus state has no leader
I20240507 08:12:02.634363 6496 raft_consensus.cc:2726] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:02.634790 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1f91718260466d99acfc9892dea536 (DEFAULT_TABLE table=default.Table_1462e4ac39014fb0aafe7485fe690e29 [id=e1821083c3ac493a8306413642160b02]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:02.657845 5250 connection.cc:535] client connection to 127.4.246.253:43459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240507 08:12:02.658020 5250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 5 similar messages]
I20240507 08:12:02.658624 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41f75cf9f00b4d03b72101f1a0668df3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.658849 5655 connection.cc:535] client connection to 127.4.246.253:43459 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240507 08:12:02.658849 5684 tablet_service.cc:1467] Processing CreateTablet for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 (DEFAULT_TABLE table=default.Table_9a8c1adf5d32474c9d849f39a54d8e84 [id=35d988e8eed14dad8783ccafffeb562f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.659752 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20240507 08:12:02.659948 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:02.660658 5841 ts_tablet_manager.cc:1888] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.660773 5841 log.cc:1196] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:12:02.661131 5841 ts_tablet_manager.cc:1909] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:02.661793 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 626) took 1338 ms. Trace:
I20240507 08:12:02.661885 5841 rpcz_store.cc:276] 0507 08:12:01.323239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.638409 (+315170us) service_pool.cc:225] Handling call
0507 08:12:02.542664 (+904255us) ts_tablet_manager.cc:1120] Deleting tablet 907b5cb0e7a54903abe0b0d0e8d617dc
0507 08:12:02.542671 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.632391 (+ 89720us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.660553 (+ 28162us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.661773 (+ 1220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":904006}
I20240507 08:12:02.662355 5841 tablet_replica.cc:330] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:02.665450 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 907b5cb0e7a54903abe0b0d0e8d617dc (table default.Table_74a9e58e257645d6b826cab9e4134f48 [id=0e277dd692e842b28638b8675da40ec9]) successfully deleted
I20240507 08:12:02.675526 5841 raft_consensus.cc:2229] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.675715 5841 raft_consensus.cc:2258] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.676328 5841 ts_tablet_manager.cc:1875] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.678453 5843 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:02.678576 5843 ts_tablet_manager.cc:1367] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.125s user 0.008s sys 0.000s
I20240507 08:12:02.679450 5843 raft_consensus.cc:348] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.679621 5843 raft_consensus.cc:374] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:02.679679 5843 raft_consensus.cc:729] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.679888 5843 consensus_queue.cc:260] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.680328 5843 ts_tablet_manager.cc:1398] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:02.680584 5843 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:02.682384 5843 tablet_bootstrap.cc:654] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:02.683568 5841 ts_tablet_manager.cc:1888] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.683665 5841 log.cc:1196] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:12:02.683990 5841 ts_tablet_manager.cc:1909] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:02.684427 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 625) took 1361 ms. Trace:
I20240507 08:12:02.684513 5841 rpcz_store.cc:276] 0507 08:12:01.322746 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.631223 (+308477us) service_pool.cc:225] Handling call
0507 08:12:02.662304 (+1031081us) ts_tablet_manager.cc:1120] Deleting tablet b22334378eec4ab3885d0f9d2d812a3e
0507 08:12:02.662315 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.679976 (+ 17661us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.683483 (+ 3507us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.684406 (+ 923us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12922,"tablet-delete.queue_time_us":1023360}
I20240507 08:12:02.684994 5841 tablet_replica.cc:330] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:02.684974 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b22334378eec4ab3885d0f9d2d812a3e (table default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]) successfully deleted
I20240507 08:12:02.685308 5841 raft_consensus.cc:2229] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.685451 5841 raft_consensus.cc:2258] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.685982 5841 ts_tablet_manager.cc:1875] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.688400 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fb9a13f3f676414d8c95abcc5889b14e" } modify_external_catalogs: true
I20240507 08:12:02.688588 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fb9a13f3f676414d8c95abcc5889b14e" } modify_external_catalogs: true
I20240507 08:12:02.689164 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b5abbb74539645c2bebcc549a43cd836"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:02.689785 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5abbb74539645c2bebcc549a43cd836 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.691325 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0612b8a20c7e43ed8606413d635f2a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:02.691839 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0612b8a20c7e43ed8606413d635f2a52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.694116 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_18d4760f5bf94ab69b1ce6418ad9507c" } new_table_name: "default.Table_cd3830ae1d3941de8d553267940b7b1a" modify_external_catalogs: true
I20240507 08:12:02.695477 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cf9647a579774e5ebc3de8cc3754fbef" } new_table_name: "default.Table_756b5368f96f42b98537f8bca05456ec" modify_external_catalogs: true
I20240507 08:12:02.698376 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:02.702924 5690 tablet_service.cc:1467] Processing CreateTablet for tablet b340567fcccb48a495033c0c430b33b0 (DEFAULT_TABLE table=default.Table_36a39b6cb68c403d8d67600517f6bb5b [id=7b5b7fee15694eabb399705704ff3017]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.704345 5841 ts_tablet_manager.cc:1888] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.704453 5841 log.cc:1196] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/193ec0b463d04835b74df80229c85cde
I20240507 08:12:02.704792 5841 ts_tablet_manager.cc:1909] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:02.705255 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 624) took 1383 ms. Trace:
I20240507 08:12:02.705338 5841 rpcz_store.cc:276] 0507 08:12:01.322001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.627557 (+305556us) service_pool.cc:225] Handling call
0507 08:12:02.684941 (+1057384us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:12:02.684953 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.694987 (+ 10034us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.704247 (+ 9260us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:02.705233 (+ 986us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1042370}
I20240507 08:12:02.705819 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 193ec0b463d04835b74df80229c85cde (table default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]) successfully deleted
I20240507 08:12:02.708144 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 627) took 1348 ms. Trace:
I20240507 08:12:02.708225 5841 rpcz_store.cc:276] 0507 08:12:01.359882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.638681 (+278799us) service_pool.cc:225] Handling call
0507 08:12:02.707996 (+1069315us) ts_tablet_manager.cc:1120] Deleting tablet 6563c42d240b4e8685dfa4972f2b9924
0507 08:12:02.708008 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.708127 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1065300}
W20240507 08:12:02.708853 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6563c42d240b4e8685dfa4972f2b9924 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6563c42d240b4e8685dfa4972f2b9924 already in progress: creating tablet
I20240507 08:12:02.709239 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 620) took 1423 ms. Trace:
I20240507 08:12:02.709316 5841 rpcz_store.cc:276] 0507 08:12:01.286030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:01.501546 (+215516us) service_pool.cc:225] Handling call
0507 08:12:02.709131 (+1207585us) ts_tablet_manager.cc:1120] Deleting tablet 4e37b73134cf42ad84b6eec90c9b805f
0507 08:12:02.709140 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.709223 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1016490}
W20240507 08:12:02.709714 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4e37b73134cf42ad84b6eec90c9b805f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e37b73134cf42ad84b6eec90c9b805f already in progress: creating tablet
I20240507 08:12:02.710281 5841 tablet_replica.cc:330] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:02.710503 5841 raft_consensus.cc:2229] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.710669 5841 raft_consensus.cc:2258] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.711237 5841 ts_tablet_manager.cc:1875] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.726043 5841 ts_tablet_manager.cc:1888] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.726168 5841 log.cc:1196] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:02.726459 5841 ts_tablet_manager.cc:1909] T 1cd02a49359743c4b4e540ab0338d8ed P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:02.733060 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1cd02a49359743c4b4e540ab0338d8ed (table default.Table_75283770e2c44851bcfa380f6385359d [id=fab372c29d234af7b0bfdae99be63919]) successfully deleted
W20240507 08:12:02.733402 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 234e582b211940aea3471e5accb9d016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e582b211940aea3471e5accb9d016 already in progress: creating tablet
W20240507 08:12:02.733691 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 62f3b6d22f7840e7b43da7a52c25a810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f3b6d22f7840e7b43da7a52c25a810 already in progress: creating tablet
W20240507 08:12:02.733918 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 767905763ba14cb8bbc52151ae9f17e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 767905763ba14cb8bbc52151ae9f17e6 already in progress: creating tablet
I20240507 08:12:02.734541 5841 tablet_replica.cc:330] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:02.734783 5841 raft_consensus.cc:2229] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.735010 5841 raft_consensus.cc:2258] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.735642 5841 ts_tablet_manager.cc:1875] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.745476 5841 ts_tablet_manager.cc:1888] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.745599 5841 log.cc:1196] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:12:02.745883 5841 ts_tablet_manager.cc:1909] T 91ee3a6c4ac642a3921560ea13bc1341 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:02.748188 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a397c40278141b7bb80ea5fcdbb125b consensus state has no leader
W20240507 08:12:02.749527 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ced879f35e14106b2609c78b7f24002 consensus state has no leader
I20240507 08:12:02.750836 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 91ee3a6c4ac642a3921560ea13bc1341 (table default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]) successfully deleted
W20240507 08:12:02.752038 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e298f11ff9f54e48b8de435f6849f39a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e298f11ff9f54e48b8de435f6849f39a already in progress: creating tablet
W20240507 08:12:02.752573 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 14929efb0ed049d6b554598a5b8cb494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14929efb0ed049d6b554598a5b8cb494 already in progress: creating tablet
I20240507 08:12:02.752808 5841 tablet_replica.cc:330] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:02.753024 5841 raft_consensus.cc:2229] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.753793 6513 ts_tablet_manager.cc:1888] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:02.753964 6513 log.cc:1196] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/916e33e40ef44d06a359b82d8fe48157
I20240507 08:12:02.754441 6513 ts_tablet_manager.cc:1909] T 916e33e40ef44d06a359b82d8fe48157 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:02.755606 6513 tablet_replica.cc:330] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:02.755888 6513 raft_consensus.cc:2229] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:02.756078 6513 raft_consensus.cc:2258] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.753175 5841 raft_consensus.cc:2258] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:02.756634 6513 ts_tablet_manager.cc:1875] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.756668 5841 ts_tablet_manager.cc:1875] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:02.757596 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.757180 6508 consensus_queue.cc:1035] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.098s
I20240507 08:12:02.759464 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:02.760490 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 124ede6467314270a2d99e0e45097c52 (DEFAULT_TABLE table=default.Table_04756d6d06eb47d2a3c61debf3d6eef1 [id=153882654fb24597a14188c78128e3bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.760905 5698 tablet_service.cc:1467] Processing CreateTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e (DEFAULT_TABLE table=default.Table_eec3f4ed076b4741aa5573b9805a83ce [id=8c0300f0ade143ad9d7681e4fe9c36fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.763636 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 128c2034d917473eb5c496319fdc9625. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.764993 5378 consensus_peers.cc:485] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:02.792315 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:12:03.077034 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240507 08:12:02.765190 5378 consensus_peers.cc:485] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:03.077559 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 312347us
W20240507 08:12:03.078042 5378 leader_election.cc:341] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.078174 5378 leader_election.cc:302] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.078627 5378 leader_election.cc:341] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.078699 6548 raft_consensus.cc:2726] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.078723 5378 leader_election.cc:302] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.078892 5378 raft_consensus.cc:2670] T 907b5cb0e7a54903abe0b0d0e8d617dc P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:03.079556 5378 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.079653 5378 leader_election.cc:302] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.079805 5378 raft_consensus.cc:2670] T b22334378eec4ab3885d0f9d2d812a3e P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:03.080372 5378 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.080463 5378 leader_election.cc:302] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.155988 5378 raft_consensus.cc:2670] T 193ec0b463d04835b74df80229c85cde P dc665657d5b34fd7aec884e0f2668e11: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:03.156641 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 76264us
I20240507 08:12:02.783878 5841 ts_tablet_manager.cc:1888] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.156913 5841 log.cc:1196] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c33710c6b55744c4ba0843c228abded3
I20240507 08:12:02.817529 5414 tablet_service.cc:1467] Processing CreateTablet for tablet d3f120d9fbf542d39508903883ecb5e6 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:03.157124 5378 leader_election.cc:341] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.157343 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f120d9fbf542d39508903883ecb5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.799047 5683 tablet_service.cc:1467] Processing CreateTablet for tablet d2240fd14155495a8c97a0af96090faf (DEFAULT_TABLE table=default.Table_a13969f58dd2425f9f93a2fadbdd178a [id=69545cc630124f44878ef9314e167dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.796574 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7007a63c36ea47c78769b5bcafec4cf9" } modify_external_catalogs: true
I20240507 08:12:02.906085 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 179e93e82ab1428a9415fb8638691074 (DEFAULT_TABLE table=default.Table_0612b8a20c7e43ed8606413d635f2a52 [id=53ea49458baf46e2b9118b3a0928d3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.157971 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7007a63c36ea47c78769b5bcafec4cf9" } modify_external_catalogs: true
I20240507 08:12:02.806934 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f (DEFAULT_TABLE table=default.Table_8c20f0267d2b49e2a564ec31f3cd52a1 [id=2df70e8274d04362b0927028ab7456d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.907975 5425 tablet_service.cc:1467] Processing CreateTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac (DEFAULT_TABLE table=default.Table_b5abbb74539645c2bebcc549a43cd836 [id=2684fec7157d4c86b0fb75daa7450886]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.808188 5696 tablet_service.cc:1467] Processing CreateTablet for tablet bc029bc110b847f79d876ec3ab568832 (DEFAULT_TABLE table=default.Table_d67b71d4e235444db255a5638a555d43 [id=54d5d5e99cf94b4c8927453711d6e0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.914959 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 039dcaf121b94ebab94ec522346d9d82 (DEFAULT_TABLE table=default.Table_96d7f5313ab8422bbfd790cea475d2ac [id=711e662558ca4542b1fbf0b4bcab74ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:02.817376 5654 leader_election.cc:341] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:03.159119 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 351332us
I20240507 08:12:02.917320 5422 tablet_service.cc:1467] Processing CreateTablet for tablet a96a6023c78f4d1196b3d2fcf2f22329 (DEFAULT_TABLE table=default.Table_fe8a85dddcfb41a18221367878c0dac7 [id=602f8fa88ad74ce085436e456c927e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.768935 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1f91718260466d99acfc9892dea536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.817521 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 63685e9ef255477db589fa572b087b57 (DEFAULT_TABLE table=default.Table_2961308b2bc546de99dfee745feced1e [id=ab07a232d652404ab199d0ce8d42f527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.159849 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b9cecc600964ee1b77098abce0e0a51. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.919948 5413 tablet_service.cc:1467] Processing CreateTablet for tablet c15763a7db9e4696b28342ea655b05f1 (DEFAULT_TABLE table=default.Table_1ba19248c0764ace86391198692f0bb5 [id=953f33502fda4426afc728321fdc3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.160122 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 690b258a9b564ed1b07aef29a774b749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.817689 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.921788 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f (DEFAULT_TABLE table=default.Table_bbc152c3898a4dbcbaa844b31b5598cb [id=923a62409ecd4e46b4b47242c8c46a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.817807 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 13fa1a807d3548d6836652379d455b0d (DEFAULT_TABLE table=default.Table_8f7c06393ac54a15842f97319969eff1 [id=7e854bf3351b4493a16b7bd7aa685baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.923138 5411 tablet_service.cc:1467] Processing CreateTablet for tablet f3004994f5944a2e9a86dbaf0627df24 (DEFAULT_TABLE table=default.Table_4929a2e8780d4124a8f206d62e5e5460 [id=649ba973c9674e08b85dd2028145586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.817919 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d (DEFAULT_TABLE table=default.Table_6cdc86ef760942ebbb3794342c9ccac7 [id=81b5c29c574a434ca6f2e6313052b635]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.924664 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff253ff827f4694bd27994825793108 (DEFAULT_TABLE table=default.Table_e667ff30b7ca4e44afc8675ec6981a0e [id=52cff27d577f48ed97d4c2c01ddaad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.818035 5687 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5fc23bfb04c2db82d1549240334e6 (DEFAULT_TABLE table=default.Table_54232a23a7cc453290c1ee7b44906b9e [id=86ce23e55d1c4ba991c506dcdc2c9414]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.937093 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 82a95b4a3799468e85051bae28c2cdd3 (DEFAULT_TABLE table=default.Table_ab3f5d9e0ac34429a34b778d025e25f1 [id=f2df62679657465382e42d6be0059885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.818162 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1f91718260466d99acfc9892dea536 (DEFAULT_TABLE table=default.Table_1462e4ac39014fb0aafe7485fe690e29 [id=e1821083c3ac493a8306413642160b02]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.941251 5843 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:02.818279 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 28e7cec189c7426795319154e4629356 (DEFAULT_TABLE table=default.Table_329152daa8064589a74946557194168f [id=23ca30ea7b534dacbb60dc2ad030ff6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.161828 5843 ts_tablet_manager.cc:1367] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.481s user 0.033s sys 0.076s
I20240507 08:12:02.766566 6497 raft_consensus.cc:482] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.161988 6497 raft_consensus.cc:504] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.162640 6497 leader_election.cc:288] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:03.162684 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b47825cbd714b91b729118e5f01eb16. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.162681 5843 raft_consensus.cc:348] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.162868 5843 raft_consensus.cc:374] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.162930 5843 raft_consensus.cc:729] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:02.818509 5720 raft_consensus.cc:2454] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:03.163128 5843 consensus_queue.cc:260] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.818607 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b340567fcccb48a495033c0c430b33b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.163623 5843 ts_tablet_manager.cc:1398] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:03.163892 5843 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.164122 5083 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskrbFV97/build/asan/bin/kudu
/tmp/dist-test-taskrbFV97/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--webserver_interface=127.4.246.253
--webserver_port=40743
--builtin_ntp_servers=127.4.246.212:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20240507 08:12:03.165352 5843 tablet_bootstrap.cc:654] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:02.797794 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4557db659d1e4591a2bffcc5077e92dd" } modify_external_catalogs: true
I20240507 08:12:03.165844 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4557db659d1e4591a2bffcc5077e92dd" } modify_external_catalogs: true
I20240507 08:12:03.166849 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab9f50b062ba42e99c471f9a70024c55. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.776435 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "077ccc3946b34008bea7b80d8aee2662" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:02.826229 6546 raft_consensus.cc:482] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.780906 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 13fa1a807d3548d6836652379d455b0d (DEFAULT_TABLE table=default.Table_8f7c06393ac54a15842f97319969eff1 [id=7e854bf3351b4493a16b7bd7aa685baf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.798103 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_64af22be42cd4e0d84bc79cfcdba4f20" } modify_external_catalogs: true
I20240507 08:12:03.167718 6546 raft_consensus.cc:504] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.167964 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fa1a807d3548d6836652379d455b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.167922 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_64af22be42cd4e0d84bc79cfcdba4f20" } modify_external_catalogs: true
I20240507 08:12:03.168377 6546 leader_election.cc:288] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:02.826927 5840 ts_tablet_manager.cc:1398] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.305s user 0.039s sys 0.053s
W20240507 08:12:03.168625 5840 ts_tablet_manager.cc:1430] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Tablet startup took 646ms
W20240507 08:12:03.168740 5840 ts_tablet_manager.cc:1432] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Trace:
0507 08:11:52.492686 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.591158 (+ 98472us) service_pool.cc:225] Handling call
0507 08:11:52.662178 (+ 71020us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:11:52.662220 (+ 42us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:11:52.682945 (+ 20725us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.711238 (+ 28293us) tablet_replica.cc:205] Creating consensus instance
0507 08:11:52.711417 (+ 179us) inbound_call.cc:173] Queueing success response
0507 08:12:01.969038 (+9257621us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0507 08:12:02.521785 (+552747us) ts_tablet_manager.cc:1399] Starting tablet replica
0507 08:12:02.521992 (+ 207us) tablet_replica.cc:259] Starting instrumentation
0507 08:12:02.522142 (+ 150us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":1373,"raft.run_cpu_time_us":78,"raft.run_wall_time_us":77,"tablet-open.queue_time_us":9257671,"thread_start_us":302321,"threads_started":1}
I20240507 08:12:02.798414 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f91e3e5674d49efa25f246c35d4f912" } modify_external_catalogs: true
I20240507 08:12:03.169212 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f91e3e5674d49efa25f246c35d4f912" } modify_external_catalogs: true
I20240507 08:12:03.169323 5840 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:02.862934 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 274947us
I20240507 08:12:03.169765 5656 leader_election.cc:302] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.170171 5656 leader_election.cc:302] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.170500 5656 leader_election.cc:302] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.170639 5656 raft_consensus.cc:2670] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:12:02.801297 5554 tablet_service.cc:1467] Processing CreateTablet for tablet f6a5fc23bfb04c2db82d1549240334e6 (DEFAULT_TABLE table=default.Table_54232a23a7cc453290c1ee7b44906b9e [id=86ce23e55d1c4ba991c506dcdc2c9414]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.170863 5840 tablet_bootstrap.cc:654] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.171154 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5fc23bfb04c2db82d1549240334e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.815609 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet e555a89d340448bc852f2441ca850213 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:02.804517 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d (DEFAULT_TABLE table=default.Table_6cdc86ef760942ebbb3794342c9ccac7 [id=81b5c29c574a434ca6f2e6313052b635]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.172737 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482e4d0e762b488b8a0cc4e81c2c821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.172817 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc4c56d06c6c4c9f83495859be8a5c5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.868530 6553 raft_consensus.cc:482] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.173133 6553 raft_consensus.cc:504] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.173750 6553 leader_election.cc:288] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.173887 6553 raft_consensus.cc:2781] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:03.173985 6553 raft_consensus.cc:686] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:03.174230 6553 consensus_queue.cc:237] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:03.175117 6508 raft_consensus.cc:2781] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:03.175314 6509 raft_consensus.cc:482] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.175395 6509 raft_consensus.cc:504] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:02.816020 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 3fce98f472ef46589dbea6b6aefa6572 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:03.175909 6509 leader_election.cc:288] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:02.806582 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 63685e9ef255477db589fa572b087b57 (DEFAULT_TABLE table=default.Table_2961308b2bc546de99dfee745feced1e [id=ab07a232d652404ab199d0ce8d42f527]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.176826 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63685e9ef255477db589fa572b087b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.177098 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33710c6b55744c4ba0843c228abded3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:02.816293 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f3bdb777b974524b5673a3f02883107 consensus state has no leader
I20240507 08:12:03.178995 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28e7cec189c7426795319154e4629356. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:02.812462 5548 tablet_service.cc:1467] Processing CreateTablet for tablet bc029bc110b847f79d876ec3ab568832 (DEFAULT_TABLE table=default.Table_d67b71d4e235444db255a5638a555d43 [id=54d5d5e99cf94b4c8927453711d6e0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.179436 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc029bc110b847f79d876ec3ab568832. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:02.841828 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 690b258a9b564ed1b07aef29a774b749 consensus state has no leader
W20240507 08:12:02.799197 5520 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 788646us
I20240507 08:12:02.843114 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776] alter complete (version 1)
I20240507 08:12:02.822870 5839 ts_tablet_manager.cc:1398] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.828s user 0.004s sys 0.007s
I20240507 08:12:02.847213 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 173a2e351dea42e9938af5edb3c5c9fe
I20240507 08:12:03.182093 6555 raft_consensus.cc:482] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.182281 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7fb0c0ec64847cb8552ebabc09e2f5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.183033 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.183588 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:02.854521 6547 tablet_replica.cc:330] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:02.857189 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 (DEFAULT_TABLE table=default.Table_630ea5aaa63747deaf53c4b3b58cf2be [id=92a8aae6d52b497599d3691d1faf1306]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.870152 5563 tablet_service.cc:1467] Processing CreateTablet for tablet d3f120d9fbf542d39508903883ecb5e6 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:02.887312 6552 raft_consensus.cc:482] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:02.901034 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 179e93e82ab1428a9415fb8638691074 (DEFAULT_TABLE table=default.Table_0612b8a20c7e43ed8606413d635f2a52 [id=53ea49458baf46e2b9118b3a0928d3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.184159 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6a5fc23bfb04c2db82d1549240334e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.184168 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:02 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.184706 6513 ts_tablet_manager.cc:1888] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.184818 6513 log.cc:1196] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:12:03.184973 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_761e0fb1290f435f82ab49c01e433ad3" } new_table_name: "default.Table_cf86bb5af528497597aa0f0affb3524e" modify_external_catalogs: true
I20240507 08:12:03.185158 6513 ts_tablet_manager.cc:1909] T 91ee3a6c4ac642a3921560ea13bc1341 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:03.185868 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:02.885699 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:03.186533 5250 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.253:43459, user_credentials={real_user=slave}} blocked reactor thread for 300810us
I20240507 08:12:03.187561 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 124ede6467314270a2d99e0e45097c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.188022 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13fa1a807d3548d6836652379d455b0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.189807 5841 ts_tablet_manager.cc:1909] T c33710c6b55744c4ba0843c228abded3 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:03.190300 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7bb96bb9d8f4930b531a3dd30d733c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.182268 6555 raft_consensus.cc:504] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:03.191835 5839 ts_tablet_manager.cc:1430] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Tablet startup took 1211ms
I20240507 08:12:03.197273 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bcd651b469bc460c8df842333fc5f367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.197815 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcd651b469bc460c8df842333fc5f367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.198925 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a6023c78f4d1196b3d2fcf2f22329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.199460 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff253ff827f4694bd27994825793108. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.199435 5262 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:03.199849 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a95b4a3799468e85051bae28c2cdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.200289 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039dcaf121b94ebab94ec522346d9d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.200465 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8615f89e816e48fc8a058e5dc466918c
I20240507 08:12:03.200649 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3004994f5944a2e9a86dbaf0627df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.201026 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.201269 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4659cc72cc3040e084a516672f35f2ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:03.201417 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179e93e82ab1428a9415fb8638691074. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.201778 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4659cc72cc3040e084a516672f35f2ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.201802 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15763a7db9e4696b28342ea655b05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.202245 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4d1fa6a8d94af982e1d23c08cbdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.203039 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e555a89d340448bc852f2441ca850213
I20240507 08:12:03.203447 5261 catalog_manager.cc:6554] ReplaceTablet: tablet e555a89d340448bc852f2441ca850213 deleted and replaced by tablet 62895377a8ac4506a2f4e10039dbdd74
I20240507 08:12:03.203670 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8615f89e816e48fc8a058e5dc466918c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.203833 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_10e42a0526054267b35b1acd7a1bc12c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.204305 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10e42a0526054267b35b1acd7a1bc12c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.205413 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3fce98f472ef46589dbea6b6aefa6572
I20240507 08:12:03.205852 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 3fce98f472ef46589dbea6b6aefa6572 deleted and replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1
I20240507 08:12:03.206185 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.206698 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.208312 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.208385 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 with delete_type TABLET_DATA_DELETED (replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:03.208962 5656 leader_election.cc:341] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:12:03.209692 5841 tablet_replica.cc:330] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:03.209875 5841 raft_consensus.cc:2229] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.210013 5841 raft_consensus.cc:2258] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.210067 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107d9b3f95e14044854ad0bf91a268ee" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:03.210127 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a199a533b4bbea3c2615acdbf95ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:03.210245 5446 raft_consensus.cc:2454] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:03.210543 5841 ts_tablet_manager.cc:1875] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:03.212383 5839 ts_tablet_manager.cc:1432] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Trace:
0507 08:11:52.251829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:11:52.252906 (+ 1077us) service_pool.cc:225] Handling call
0507 08:11:52.264547 (+ 11641us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:11:52.264571 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:11:52.491612 (+227041us) tablet_metadata.cc:641] Metadata flushed
0507 08:11:52.606315 (+114703us) tablet_replica.cc:205] Creating consensus instance
0507 08:11:52.606457 (+ 142us) inbound_call.cc:173] Queueing success response
0507 08:12:01.942491 (+9336034us) ts_tablet_manager.cc:1343] Bootstrapping tablet
0507 08:12:01.970787 (+ 28296us) ts_tablet_manager.cc:1399] Starting tablet replica
0507 08:12:01.970952 (+ 165us) tablet_replica.cc:259] Starting instrumentation
0507 08:12:01.971089 (+ 137us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":16221,"raft.run_cpu_time_us":63,"raft.run_wall_time_us":63,"tablet-open.queue_time_us":9336064,"thread_start_us":159037,"threads_started":1}
I20240507 08:12:03.213187 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2240fd14155495a8c97a0af96090faf. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.213225 5520 leader_election.cc:341] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 8ed8bac124cb4de983f0fdbaf728975f
I20240507 08:12:03.213459 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c7d19b7cc39a4337826819a20205aecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.215647 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 35e34b60b78b402faf51e003c97db787
I20240507 08:12:03.216233 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8497f8ea3c9a48d281d91c90ea305cb2
I20240507 08:12:03.216384 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_15e48d810d164ffba2efc7a11fc51244" } new_table_name: "default.Table_11758edf8d62474b993d1527d5d04ce4" modify_external_catalogs: true
W20240507 08:12:03.217408 5656 leader_election.cc:341] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.217621 5656 leader_election.cc:302] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:03.217941 6546 raft_consensus.cc:2781] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:03.218017 6546 raft_consensus.cc:482] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:03.218070 6546 raft_consensus.cc:3037] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:03.218179 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b340567fcccb48a495033c0c430b33b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.190318 6552 raft_consensus.cc:504] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:03.218305 5839 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.224186 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_746e0a45757a4d1e898c096f11542589" } new_table_name: "default.Table_9383fc310d0f4a6fb20a0c555d0e8257" modify_external_catalogs: true
I20240507 08:12:03.226855 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:03.234017 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9774c61bef464f96ab38fafb81f92ca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.234568 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9774c61bef464f96ab38fafb81f92ca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.235229 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c33710c6b55744c4ba0843c228abded3 (table default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]) successfully deleted
W20240507 08:12:03.236043 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 173a2e351dea42e9938af5edb3c5c9fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 173a2e351dea42e9938af5edb3c5c9fe already in progress: creating tablet
W20240507 08:12:03.236685 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 79138us
I20240507 08:12:03.242489 6547 raft_consensus.cc:2229] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.244009 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f120d9fbf542d39508903883ecb5e6. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.243294 5517 leader_election.cc:341] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 8ed8bac124cb4de983f0fdbaf728975f
I20240507 08:12:03.245766 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3dc5823424924e0bbff545ee187770d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.246968 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5aca2077b76b49e3bf9daae57dc09b8f consensus state has no leader
I20240507 08:12:03.247494 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b17843ca835c4017a73c4bf3baf9afea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.248072 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b17843ca835c4017a73c4bf3baf9afea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:03.249827 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dc5823424924e0bbff545ee187770d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.251608 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9d069145365640e589b486e39c72d4a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:03.251668 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6d4810aabea24b539d53b0e43c7298ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.252079 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d069145365640e589b486e39c72d4a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:03.252080 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d4810aabea24b539d53b0e43c7298ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:03.254637 5520 leader_election.cc:341] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 191adef34efa4c3999d4642a80177294
I20240507 08:12:03.256675 5517 leader_election.cc:302] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.269160 5843 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.269313 5843 ts_tablet_manager.cc:1367] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.105s user 0.005s sys 0.005s
I20240507 08:12:03.270131 5843 raft_consensus.cc:348] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.270324 5843 raft_consensus.cc:374] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.270381 5843 raft_consensus.cc:729] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.270584 5843 consensus_queue.cc:260] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.271092 5843 ts_tablet_manager.cc:1398] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:03.272055 5843 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.273922 5843 tablet_bootstrap.cc:654] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.280508 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b14c80b96bb43088e1d1354a1dbdd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.291404 5843 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.291559 5843 ts_tablet_manager.cc:1367] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20240507 08:12:03.292454 5843 raft_consensus.cc:348] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.292685 5843 raft_consensus.cc:374] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.292753 5843 raft_consensus.cc:729] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.293006 5843 consensus_queue.cc:260] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.293650 5843 ts_tablet_manager.cc:1398] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 08:12:03.293967 5843 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.296131 5843 tablet_bootstrap.cc:654] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:03.313583 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06898414c5494cd380e0c932f4589014 consensus state has no leader
I20240507 08:12:03.314296 5843 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.314463 5843 ts_tablet_manager.cc:1367] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20240507 08:12:03.337841 5843 raft_consensus.cc:348] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.338105 5843 raft_consensus.cc:374] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.338171 5843 raft_consensus.cc:729] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.338390 5843 consensus_queue.cc:260] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.339031 5843 ts_tablet_manager.cc:1398] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240507 08:12:03.352073 6508 raft_consensus.cc:686] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:03.352450 6508 consensus_queue.cc:237] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.353087 6547 raft_consensus.cc:2258] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.351526 6552 leader_election.cc:288] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:03.353364 6555 leader_election.cc:288] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.357806 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 482e4d0e762b488b8a0cc4e81c2c821d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.358050 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d7d8bcb635416693395b1f9ad27e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.358206 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2240fd14155495a8c97a0af96090faf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.358058 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 643) took 1271 ms. Trace:
I20240507 08:12:03.359584 6513 rpcz_store.cc:276] 0507 08:12:02.087031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.087068 (+ 37us) service_pool.cc:225] Handling call
0507 08:12:02.755495 (+668427us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:12:02.755508 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:02.819516 (+ 64008us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.184581 (+365065us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.358026 (+173445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":663581}
I20240507 08:12:03.360905 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.362664 6552 raft_consensus.cc:2726] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.365422 5520 leader_election.cc:341] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:12:03.366896 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 639) took 1288 ms. Trace:
I20240507 08:12:03.367017 6513 rpcz_store.cc:276] 0507 08:12:02.077954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.078663 (+ 709us) service_pool.cc:225] Handling call
0507 08:12:03.359909 (+1281246us) ts_tablet_manager.cc:1120] Deleting tablet e298f11ff9f54e48b8de435f6849f39a
0507 08:12:03.359923 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.366875 (+ 6952us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":850220}
I20240507 08:12:03.368022 5839 tablet_bootstrap.cc:654] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.370049 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1f91718260466d99acfc9892dea536. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.371019 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179e93e82ab1428a9415fb8638691074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.371271 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96d7d8bcb635416693395b1f9ad27e2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.373545 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:03.374550 5378 leader_election.cc:341] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.374673 6547 ts_tablet_manager.cc:1875] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.376209 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.376955 5843 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.380764 5843 tablet_bootstrap.cc:654] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.382655 5559 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 636) took 1038 ms. Trace:
I20240507 08:12:03.382766 5559 rpcz_store.cc:276] 0507 08:12:02.343824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.795735 (+451911us) service_pool.cc:225] Handling call
0507 08:12:03.172509 (+376774us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.172535 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.175605 (+ 3070us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.381803 (+206198us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.381950 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:12:03.384657 5520 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 99711.6us
I20240507 08:12:03.385707 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.383316 5559 tablet_service.cc:1467] Processing CreateTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac (DEFAULT_TABLE table=default.Table_b5abbb74539645c2bebcc549a43cd836 [id=2684fec7157d4c86b0fb75daa7450886]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:03.396960 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 156765us
I20240507 08:12:03.401506 6552 raft_consensus.cc:482] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.404660 5555 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 616) took 1297 ms. Trace:
I20240507 08:12:03.405822 5555 rpcz_store.cc:276] 0507 08:12:02.101958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103317 (+ 1359us) service_pool.cc:225] Handling call
0507 08:12:02.523383 (+420066us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.523406 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.163407 (+640001us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.396152 (+232745us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.399078 (+ 2926us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.406986 6552 raft_consensus.cc:504] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.408948 6552 leader_election.cc:288] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.409832 5557 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 614) took 1306 ms. Trace:
I20240507 08:12:03.409951 5557 rpcz_store.cc:276] 0507 08:12:02.101716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103010 (+ 1294us) service_pool.cc:225] Handling call
0507 08:12:02.523724 (+420714us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.523754 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.183274 (+659520us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.404357 (+221083us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.408560 (+ 4203us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20240507 08:12:03.413664 5520 leader_election.cc:341] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5e53409fe13e4a9585dbba648a25cc07
I20240507 08:12:03.415211 5565 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 611) took 1318 ms. Trace:
I20240507 08:12:03.415519 5565 rpcz_store.cc:276] 0507 08:12:02.096632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.096986 (+ 354us) service_pool.cc:225] Handling call
0507 08:12:02.522477 (+425491us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.522497 (+ 20us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.199331 (+676834us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.415034 (+215703us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.415175 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.427213 5554 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 635) took 1083 ms. Trace:
I20240507 08:12:03.427491 5554 rpcz_store.cc:276] 0507 08:12:02.343678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.791723 (+448045us) service_pool.cc:225] Handling call
0507 08:12:03.170934 (+379211us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.170959 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.192120 (+ 21161us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.426989 (+234869us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.427169 (+ 180us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.430200 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63685e9ef255477db589fa572b087b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.430514 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc029bc110b847f79d876ec3ab568832. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.433826 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b14c80b96bb43088e1d1354a1dbdd31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.438223 5377 leader_election.cc:302] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.439270 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 179e93e82ab1428a9415fb8638691074 (DEFAULT_TABLE table=default.Table_0612b8a20c7e43ed8606413d635f2a52 [id=53ea49458baf46e2b9118b3a0928d3be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.439363 5547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 634) took 1104 ms. Trace:
I20240507 08:12:03.439493 5547 rpcz_store.cc:276] 0507 08:12:02.334734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.777720 (+442986us) service_pool.cc:225] Handling call
0507 08:12:03.167730 (+390010us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.167779 (+ 49us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.170769 (+ 2990us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.439182 (+268413us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.439324 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.439646 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 179e93e82ab1428a9415fb8638691074. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.439699 6546 raft_consensus.cc:504] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.440270 6546 leader_election.cc:288] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.441845 5689 tablet_service.cc:1467] Processing CreateTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac (DEFAULT_TABLE table=default.Table_b5abbb74539645c2bebcc549a43cd836 [id=2684fec7157d4c86b0fb75daa7450886]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.442188 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4d1fa6a8d94af982e1d23c08cbdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.442795 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 039dcaf121b94ebab94ec522346d9d82 (DEFAULT_TABLE table=default.Table_96d7f5313ab8422bbfd790cea475d2ac [id=711e662558ca4542b1fbf0b4bcab74ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.443194 5700 tablet_service.cc:1467] Processing CreateTablet for tablet a96a6023c78f4d1196b3d2fcf2f22329 (DEFAULT_TABLE table=default.Table_fe8a85dddcfb41a18221367878c0dac7 [id=602f8fa88ad74ce085436e456c927e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.443616 5695 tablet_service.cc:1467] Processing CreateTablet for tablet c15763a7db9e4696b28342ea655b05f1 (DEFAULT_TABLE table=default.Table_1ba19248c0764ace86391198692f0bb5 [id=953f33502fda4426afc728321fdc3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.443997 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f (DEFAULT_TABLE table=default.Table_bbc152c3898a4dbcbaa844b31b5598cb [id=923a62409ecd4e46b4b47242c8c46a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.444267 6548 raft_consensus.cc:2781] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:03.445502 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 916e33e40ef44d06a359b82d8fe48157 (table default.Table_6ad0c511f95b4b09abad3bc4a13d1238 [id=14589abdd357409cbe9837a33208e00d]) successfully deleted
I20240507 08:12:03.445809 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 91ee3a6c4ac642a3921560ea13bc1341 (table default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]) successfully deleted
I20240507 08:12:03.445935 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 82a95b4a3799468e85051bae28c2cdd3 (DEFAULT_TABLE table=default.Table_ab3f5d9e0ac34429a34b778d025e25f1 [id=f2df62679657465382e42d6be0059885]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:03.446067 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e298f11ff9f54e48b8de435f6849f39a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e298f11ff9f54e48b8de435f6849f39a already in progress: creating tablet
I20240507 08:12:03.446702 5840 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:03.446820 5840 ts_tablet_manager.cc:1367] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.278s user 0.019s sys 0.011s
I20240507 08:12:03.447320 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973a199a533b4bbea3c2615acdbf95ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.447551 5840 raft_consensus.cc:348] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.447711 5840 raft_consensus.cc:374] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.447764 5840 raft_consensus.cc:729] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.447899 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c33710c6b55744c4ba0843c228abded3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.447959 5840 consensus_queue.cc:260] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.448282 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107d9b3f95e14044854ad0bf91a268ee" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.449513 5840 ts_tablet_manager.cc:1398] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:03.450582 5840 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:03.450743 5560 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 627) took 1148 ms. Trace:
I20240507 08:12:03.450870 5560 rpcz_store.cc:276] 0507 08:12:02.301784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.301835 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:02.552960 (+251125us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.552990 (+ 30us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.442181 (+889191us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.450574 (+ 8393us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.450707 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.451467 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.451516 5556 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 617) took 1349 ms. Trace:
I20240507 08:12:03.451632 5556 rpcz_store.cc:276] 0507 08:12:02.102064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103355 (+ 1291us) service_pool.cc:225] Handling call
0507 08:12:02.552673 (+449318us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.552705 (+ 32us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.165491 (+612786us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.451356 (+285865us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.451478 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.451731 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a6023c78f4d1196b3d2fcf2f22329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.452186 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15763a7db9e4696b28342ea655b05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.452618 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039dcaf121b94ebab94ec522346d9d82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.452611 5682 tablet_service.cc:1467] Processing CreateTablet for tablet f3004994f5944a2e9a86dbaf0627df24 (DEFAULT_TABLE table=default.Table_4929a2e8780d4124a8f206d62e5e5460 [id=649ba973c9674e08b85dd2028145586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.455083 5550 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 631) took 1130 ms. Trace:
I20240507 08:12:03.455193 5550 rpcz_store.cc:276] 0507 08:12:02.324516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.551549 (+227033us) service_pool.cc:225] Handling call
0507 08:12:02.553124 (+ 1575us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.553145 (+ 21us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.446234 (+893089us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.454896 (+ 8662us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.455038 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.457669 5546 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 633) took 1131 ms. Trace:
I20240507 08:12:03.459604 5840 tablet_bootstrap.cc:654] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.460847 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:03.460989 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff253ff827f4694bd27994825793108 (DEFAULT_TABLE table=default.Table_e667ff30b7ca4e44afc8675ec6981a0e [id=52cff27d577f48ed97d4c2c01ddaad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.469203 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:02 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.472594 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a95b4a3799468e85051bae28c2cdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.476950 5841 ts_tablet_manager.cc:1888] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.477973 5841 log.cc:1196] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/973a199a533b4bbea3c2615acdbf95ca
I20240507 08:12:03.481623 5841 ts_tablet_manager.cc:1909] T 973a199a533b4bbea3c2615acdbf95ca P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:03.485183 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 973a199a533b4bbea3c2615acdbf95ca (table default.Table_964368f3c7824528b36f3fb37bb2e215 [id=4ecddc64b3da4de5a70e7a211f9ea037]) successfully deleted
I20240507 08:12:03.486845 5841 tablet_replica.cc:330] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:03.488745 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.488925 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 with delete_type TABLET_DATA_DELETED (replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.489046 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.489382 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff253ff827f4694bd27994825793108. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.489060 5841 raft_consensus.cc:2229] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:03.492066 5841 raft_consensus.cc:2258] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.495025 5546 rpcz_store.cc:276] 0507 08:12:02.326196 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.628172 (+301976us) service_pool.cc:225] Handling call
0507 08:12:02.635192 (+ 7020us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.635982 (+ 790us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.395667 (+759685us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.457462 (+ 61795us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.457614 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.464437 5551 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 626) took 1176 ms. Trace:
I20240507 08:12:03.495633 5551 rpcz_store.cc:276] 0507 08:12:02.287863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.287932 (+ 69us) service_pool.cc:225] Handling call
0507 08:12:02.552878 (+264946us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.552903 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.399463 (+846560us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.464276 (+ 64813us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.464399 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.477103 5558 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 639) took 1128 ms. Trace:
I20240507 08:12:03.496150 5558 rpcz_store.cc:276] 0507 08:12:02.348226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.831396 (+483170us) service_pool.cc:225] Handling call
0507 08:12:03.184006 (+352610us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.184032 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.470854 (+286822us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.476894 (+ 6040us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.477066 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.480011 5552 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 637) took 1136 ms. Trace:
I20240507 08:12:03.496671 5552 rpcz_store.cc:276] 0507 08:12:02.343971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.806361 (+462390us) service_pool.cc:225] Handling call
0507 08:12:03.176638 (+370277us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.176665 (+ 27us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.373905 (+197240us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.479831 (+105926us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.479973 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.482894 5562 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 619) took 1380 ms. Trace:
I20240507 08:12:03.497290 5562 rpcz_store.cc:276] 0507 08:12:02.102309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103446 (+ 1137us) service_pool.cc:225] Handling call
0507 08:12:02.552791 (+449345us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.552817 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.413192 (+860375us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.482725 (+ 69533us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.482858 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.487656 5553 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 615) took 1385 ms. Trace:
I20240507 08:12:03.497771 5553 rpcz_store.cc:276] 0507 08:12:02.101844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103285 (+ 1441us) service_pool.cc:225] Handling call
0507 08:12:02.574183 (+470898us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.574214 (+ 31us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.381394 (+807180us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.487495 (+106101us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.487608 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.489154 5548 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 638) took 1141 ms. Trace:
I20240507 08:12:03.498395 5548 rpcz_store.cc:276] 0507 08:12:02.348089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.811798 (+463709us) service_pool.cc:225] Handling call
0507 08:12:03.179248 (+367450us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.179274 (+ 26us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.214795 (+ 35521us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.489000 (+274205us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.489119 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.498118 5841 ts_tablet_manager.cc:1875] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.491487 5564 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 630) took 1167 ms. Trace:
I20240507 08:12:03.498971 5564 rpcz_store.cc:276] 0507 08:12:02.323932 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.524415 (+200483us) service_pool.cc:225] Handling call
0507 08:12:02.581499 (+ 57084us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.581523 (+ 24us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.484550 (+903027us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.491310 (+ 6760us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.491439 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.469254 5561 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37096 (request call id 612) took 1372 ms. Trace:
I20240507 08:12:03.499496 5561 rpcz_store.cc:276] 0507 08:12:02.096741 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.097014 (+ 273us) service_pool.cc:225] Handling call
0507 08:12:02.522987 (+425973us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:02.523010 (+ 23us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.466412 (+943402us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.469094 (+ 2682us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.469221 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.501698 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 8615f89e816e48fc8a058e5dc466918c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.502286 5555 tablet_service.cc:1467] Processing CreateTablet for tablet c15763a7db9e4696b28342ea655b05f1 (DEFAULT_TABLE table=default.Table_1ba19248c0764ace86391198692f0bb5 [id=953f33502fda4426afc728321fdc3347]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.502749 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f (DEFAULT_TABLE table=default.Table_bbc152c3898a4dbcbaa844b31b5598cb [id=923a62409ecd4e46b4b47242c8c46a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.502704 5557 tablet_service.cc:1467] Processing CreateTablet for tablet a96a6023c78f4d1196b3d2fcf2f22329 (DEFAULT_TABLE table=default.Table_fe8a85dddcfb41a18221367878c0dac7 [id=602f8fa88ad74ce085436e456c927e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.503250 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd4d1fa6a8d94af982e1d23c08cbdeac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.505610 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c15763a7db9e4696b28342ea655b05f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.505610 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3004994f5944a2e9a86dbaf0627df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.506537 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 9ff253ff827f4694bd27994825793108 (DEFAULT_TABLE table=default.Table_e667ff30b7ca4e44afc8675ec6981a0e [id=52cff27d577f48ed97d4c2c01ddaad37]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.507055 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ff253ff827f4694bd27994825793108. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.508483 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 82a95b4a3799468e85051bae28c2cdd3 (DEFAULT_TABLE table=default.Table_ab3f5d9e0ac34429a34b778d025e25f1 [id=f2df62679657465382e42d6be0059885]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.508881 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82a95b4a3799468e85051bae28c2cdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.510489 5684 tablet_service.cc:1467] Processing CreateTablet for tablet d3f120d9fbf542d39508903883ecb5e6 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.511271 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a96a6023c78f4d1196b3d2fcf2f22329. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.513625 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 173a2e351dea42e9938af5edb3c5c9fe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:02 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.513895 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 8615f89e816e48fc8a058e5dc466918c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.514282 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.514617 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 with delete_type TABLET_DATA_DELETED (replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.514928 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.515461 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.515239 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.515873 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.502328 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 039dcaf121b94ebab94ec522346d9d82 (DEFAULT_TABLE table=default.Table_96d7f5313ab8422bbfd790cea475d2ac [id=711e662558ca4542b1fbf0b4bcab74ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.494577 6547 ts_tablet_manager.cc:1888] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.535985 5843 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.536643 5843 ts_tablet_manager.cc:1367] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.020s
W20240507 08:12:03.538363 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80c892dab4414411acce4dd160079672 consensus state has no leader
I20240507 08:12:03.539945 5843 raft_consensus.cc:348] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.541170 5843 raft_consensus.cc:374] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.541616 5843 raft_consensus.cc:729] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.542560 5843 consensus_queue.cc:260] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.543812 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_910aab0a8d6a4249b5e8a0ae88916c9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.544296 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 9ced879f35e14106b2609c78b7f24002 (table default.Table_dea0e2b1b090454cbd15a00f7e792d6e [id=aa09d786696945d5b4060819737a6e11], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9ced879f35e14106b2609c78b7f24002 consensus state has no leader
I20240507 08:12:03.544672 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f8765d860074430f9fb9ecfba92d9dd7" } modify_external_catalogs: true
I20240507 08:12:03.544741 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bfe2378493e64efb93444e140f410dfe" } modify_external_catalogs: true
I20240507 08:12:03.544889 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bfe2378493e64efb93444e140f410dfe" } modify_external_catalogs: true
I20240507 08:12:03.545080 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a58f4e04c4147368d43d162c9613c9f" } modify_external_catalogs: true
I20240507 08:12:03.545233 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a58f4e04c4147368d43d162c9613c9f" } modify_external_catalogs: true
I20240507 08:12:03.545419 5843 ts_tablet_manager.cc:1398] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240507 08:12:03.545600 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12604f6607614bb6a9001e40b82cabe3" } new_table_name: "default.Table_bd4ed42b3028472ab247218a0e87e059" modify_external_catalogs: true
W20240507 08:12:03.545888 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_910aab0a8d6a4249b5e8a0ae88916c9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.544893 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f8765d860074430f9fb9ecfba92d9dd7" } modify_external_catalogs: true
I20240507 08:12:03.546375 5843 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.548884 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6549c56b83e24fe49342c6864c56162c" } new_table_name: "default.Table_28e3986bcadb49cb861bd00e92c44f6b" modify_external_catalogs: true
I20240507 08:12:03.549378 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_601dd7f065e343eba100d1a64f0a72a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.550011 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_601dd7f065e343eba100d1a64f0a72a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.551697 5843 tablet_bootstrap.cc:654] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.552516 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bf6fa5a914c14039a8f8940d2197f70a" } modify_external_catalogs: true
I20240507 08:12:03.552695 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bf6fa5a914c14039a8f8940d2197f70a" } modify_external_catalogs: true
I20240507 08:12:03.553853 5841 ts_tablet_manager.cc:1888] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:03.553964 5841 log.cc:1196] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8615f89e816e48fc8a058e5dc466918c
I20240507 08:12:03.554236 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet b2de528d9fc046f183f70388cffab6c8 (table default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a]). Expected version 2 got 0
I20240507 08:12:03.554430 5841 ts_tablet_manager.cc:1909] T 8615f89e816e48fc8a058e5dc466918c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:03.555114 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8615f89e816e48fc8a058e5dc466918c (table default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]) successfully deleted
I20240507 08:12:03.555593 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_981d958d70ea464e8dd9a203a2374687" } modify_external_catalogs: true
W20240507 08:12:03.555688 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e555a89d340448bc852f2441ca850213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e555a89d340448bc852f2441ca850213 already in progress: creating tablet
I20240507 08:12:03.555846 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_981d958d70ea464e8dd9a203a2374687" } modify_external_catalogs: true
W20240507 08:12:03.558495 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3fce98f472ef46589dbea6b6aefa6572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fce98f472ef46589dbea6b6aefa6572 already in progress: creating tablet
W20240507 08:12:03.558784 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 35e34b60b78b402faf51e003c97db787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e34b60b78b402faf51e003c97db787 already in progress: creating tablet
W20240507 08:12:03.559612 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1037c3c0af5e464bb23a311e552fcf21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1037c3c0af5e464bb23a311e552fcf21 already in progress: creating tablet
I20240507 08:12:03.560091 6547 log.cc:1196] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/191adef34efa4c3999d4642a80177294
I20240507 08:12:03.561228 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.561550 6547 ts_tablet_manager.cc:1909] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:03.561939 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 618) took 1459 ms. Trace:
I20240507 08:12:03.562069 6547 rpcz_store.cc:276] 0507 08:12:02.102200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103383 (+ 1183us) service_pool.cc:225] Handling call
0507 08:12:02.839735 (+736352us) ts_tablet_manager.cc:1120] Deleting tablet 191adef34efa4c3999d4642a80177294
0507 08:12:02.840786 (+ 1051us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.460290 (+619504us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.494436 (+ 34146us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.561893 (+ 67457us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33598,"tablet-delete.queue_time_us":311735,"thread_start_us":304117,"threads_started":1}
I20240507 08:12:03.562642 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 623) took 1459 ms. Trace:
I20240507 08:12:03.562772 6547 rpcz_store.cc:276] 0507 08:12:02.102736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103560 (+ 824us) service_pool.cc:225] Handling call
0507 08:12:03.562497 (+1458937us) ts_tablet_manager.cc:1120] Deleting tablet 62f3b6d22f7840e7b43da7a52c25a810
0507 08:12:03.562506 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.562609 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038312}
I20240507 08:12:03.563148 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 620) took 1460 ms. Trace:
I20240507 08:12:03.563261 6547 rpcz_store.cc:276] 0507 08:12:02.102407 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103475 (+ 1068us) service_pool.cc:225] Handling call
0507 08:12:03.563005 (+1459530us) ts_tablet_manager.cc:1120] Deleting tablet 14929efb0ed049d6b554598a5b8cb494
0507 08:12:03.563014 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.563111 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038687}
I20240507 08:12:03.563633 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 622) took 1461 ms. Trace:
I20240507 08:12:03.563740 6547 rpcz_store.cc:276] 0507 08:12:02.102632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103533 (+ 901us) service_pool.cc:225] Handling call
0507 08:12:03.563497 (+1459964us) ts_tablet_manager.cc:1120] Deleting tablet 234e582b211940aea3471e5accb9d016
0507 08:12:03.563506 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.563598 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039087}
I20240507 08:12:03.564126 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 624) took 1461 ms. Trace:
I20240507 08:12:03.564242 6547 rpcz_store.cc:276] 0507 08:12:02.102836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103585 (+ 749us) service_pool.cc:225] Handling call
0507 08:12:03.564000 (+1460415us) ts_tablet_manager.cc:1120] Deleting tablet 767905763ba14cb8bbc52151ae9f17e6
0507 08:12:03.564011 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.564094 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1012515}
I20240507 08:12:03.564589 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 621) took 1462 ms. Trace:
I20240507 08:12:03.564703 6547 rpcz_store.cc:276] 0507 08:12:02.102515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.103503 (+ 988us) service_pool.cc:225] Handling call
0507 08:12:03.564470 (+1460967us) ts_tablet_manager.cc:1120] Deleting tablet e298f11ff9f54e48b8de435f6849f39a
0507 08:12:03.564480 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:03.564558 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1012284}
I20240507 08:12:03.568070 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:03.568601 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039dcaf121b94ebab94ec522346d9d82. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.570538 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.571667 5550 tablet_service.cc:1467] Processing CreateTablet for tablet f3004994f5944a2e9a86dbaf0627df24 (DEFAULT_TABLE table=default.Table_4929a2e8780d4124a8f206d62e5e5460 [id=649ba973c9674e08b85dd2028145586f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.573549 5839 tablet_bootstrap.cc:492] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:03.574323 5839 ts_tablet_manager.cc:1367] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.361s user 0.005s sys 0.007s
I20240507 08:12:03.576081 5839 raft_consensus.cc:348] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.576316 5839 raft_consensus.cc:374] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.576402 5839 raft_consensus.cc:729] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.576643 5839 consensus_queue.cc:260] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.577176 5839 ts_tablet_manager.cc:1398] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240507 08:12:03.583803 5841 tablet_replica.cc:330] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:03.584003 5841 raft_consensus.cc:2229] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.584131 5841 raft_consensus.cc:2258] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:03.584123 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8497f8ea3c9a48d281d91c90ea305cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8497f8ea3c9a48d281d91c90ea305cb2 already in progress: creating tablet
I20240507 08:12:03.584605 5841 ts_tablet_manager.cc:1875] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.584749 6548 raft_consensus.cc:686] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:03.585069 6548 consensus_queue.cc:237] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.591796 5843 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.592063 5843 ts_tablet_manager.cc:1367] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.007s
I20240507 08:12:03.593502 5843 raft_consensus.cc:348] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.593830 5843 raft_consensus.cc:374] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.593915 5843 raft_consensus.cc:729] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.594210 5843 consensus_queue.cc:260] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.595268 5843 ts_tablet_manager.cc:1398] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:12:03.596186 5843 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.597735 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3004994f5944a2e9a86dbaf0627df24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.600394 5839 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.600641 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3f120d9fbf542d39508903883ecb5e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.602216 5843 tablet_bootstrap.cc:654] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.607237 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107d9b3f95e14044854ad0bf91a268ee" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:03.608464 6567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c202eef7f3466fa3e703644b40879d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:12:03.608757 6567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d [sys.catalog]: This master's current role is: LEADER
W20240507 08:12:03.609748 5656 leader_election.cc:341] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:12:03.611507 5545 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 123) took 1411 ms. Trace:
I20240507 08:12:03.612526 5545 rpcz_store.cc:276] 0507 08:12:02.199950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.200011 (+ 61us) service_pool.cc:225] Handling call
0507 08:12:03.255996 (+1055985us) spinlock_profiling.cc:243] Waited 91.6 ms on lock 0x6160000c5254. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2af0cb00 00007f2f2ec1e95f 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58 0000000000a60bcc
0507 08:12:03.611453 (+355457us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":210626432}
I20240507 08:12:03.614033 5841 ts_tablet_manager.cc:1888] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.614127 5841 log.cc:1196] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/107d9b3f95e14044854ad0bf91a268ee
I20240507 08:12:03.614379 5841 ts_tablet_manager.cc:1909] T 107d9b3f95e14044854ad0bf91a268ee P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:03.615271 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 107d9b3f95e14044854ad0bf91a268ee (table default.Table_351291351f3c4972ac5f9cd896d68312 [id=ccea3f017cc84cce95f3010023a76bb4]) successfully deleted
W20240507 08:12:03.618547 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 179667us
W20240507 08:12:03.625370 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fc65070bbae4db19c5d14b6670d599c consensus state has no leader
I20240507 08:12:03.627589 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 648113d9b8164cc59ba238ace8f65b39 (DEFAULT_TABLE table=default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5 [id=cbd530b6bd504bf48785968ba2ac3f75]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:03.627854 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1944cb716e7b498a8f13a78b14d8c933 consensus state has no leader
I20240507 08:12:03.627977 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648113d9b8164cc59ba238ace8f65b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.629935 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9d069145365640e589b486e39c72d4a0" } new_table_name: "default.Table_b2afebeaafe84471bf8b1ae310b46036" modify_external_catalogs: true
I20240507 08:12:03.631117 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_55ca0ba456d0447d95c4f2df7eee0591" } new_table_name: "default.Table_4205c8d3d874424599f4c1e843ac9543" modify_external_catalogs: true
I20240507 08:12:03.632522 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b5eee064314f4431b951651d05ed1e13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.633126 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5eee064314f4431b951651d05ed1e13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.634199 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.634541 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62895377a8ac4506a2f4e10039dbdd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.634426 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_44802e7664e8479c97e888d0a2b4dbf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.634889 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44802e7664e8479c97e888d0a2b4dbf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.635659 5839 tablet_bootstrap.cc:654] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.637831 5410 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb720c09c5486995391837fc54e7ce (DEFAULT_TABLE table=default.Table_bcd651b469bc460c8df842333fc5f367 [id=b07a3851b8cd427ba7b48f9be771544d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.638170 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb720c09c5486995391837fc54e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.643074 5411 tablet_service.cc:1467] Processing CreateTablet for tablet bf8fe5157dd9491cac7e0237c3d39ec1 (DEFAULT_TABLE table=default.Table_a48b5e1d22504c999c0df606f5092147 [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.643383 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8fe5157dd9491cac7e0237c3d39ec1. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.644400 5654 consensus_peers.cc:485] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:03.644583 5654 leader_election.cc:341] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.644670 5654 leader_election.cc:302] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:03.645166 5654 leader_election.cc:341] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.645246 5654 leader_election.cc:302] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.645805 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 (DEFAULT_TABLE table=default.Table_10e42a0526054267b35b1acd7a1bc12c [id=da75815357824bd3a59216d34246edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.646080 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.646133 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 648113d9b8164cc59ba238ace8f65b39 (DEFAULT_TABLE table=default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5 [id=cbd530b6bd504bf48785968ba2ac3f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.646435 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648113d9b8164cc59ba238ace8f65b39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.649320 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 182124e13ca749c8a2a96d38514158b1 (DEFAULT_TABLE table=default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f [id=4e15b2c7c70f426fa80aa0c92a00e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.649732 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182124e13ca749c8a2a96d38514158b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.650794 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d (DEFAULT_TABLE table=default.Table_4659cc72cc3040e084a516672f35f2ec [id=c0a82997be2d45cca329a564fa01150b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.651089 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ed504c4cf443e48ab285466b9e9b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.652451 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 182124e13ca749c8a2a96d38514158b1 (DEFAULT_TABLE table=default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f [id=4e15b2c7c70f426fa80aa0c92a00e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.652765 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182124e13ca749c8a2a96d38514158b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.653792 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d (DEFAULT_TABLE table=default.Table_4659cc72cc3040e084a516672f35f2ec [id=c0a82997be2d45cca329a564fa01150b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.653903 6546 raft_consensus.cc:2726] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.654156 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ed504c4cf443e48ab285466b9e9b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.654222 6546 raft_consensus.cc:2726] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.654399 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "107d9b3f95e14044854ad0bf91a268ee" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:03.655021 5544 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 122) took 1537 ms. Trace:
I20240507 08:12:03.655109 5544 rpcz_store.cc:276] 0507 08:12:02.117496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.117888 (+ 392us) service_pool.cc:225] Handling call
0507 08:12:03.654997 (+1537109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":768}
W20240507 08:12:03.656229 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 173a2e351dea42e9938af5edb3c5c9fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 173a2e351dea42e9938af5edb3c5c9fe already in progress: creating tablet
W20240507 08:12:03.645531 5654 leader_election.cc:341] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:03.659870 5377 leader_election.cc:341] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.661921 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.662669 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 8497f8ea3c9a48d281d91c90ea305cb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.663643 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:03.663839 5654 leader_election.cc:341] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.663933 5654 leader_election.cc:302] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.664184 6609 raft_consensus.cc:2726] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.670425 5698 tablet_service.cc:1467] Processing CreateTablet for tablet bf8fe5157dd9491cac7e0237c3d39ec1 (DEFAULT_TABLE table=default.Table_a48b5e1d22504c999c0df606f5092147 [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.670795 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8fe5157dd9491cac7e0237c3d39ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.671327 5691 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb720c09c5486995391837fc54e7ce (DEFAULT_TABLE table=default.Table_bcd651b469bc460c8df842333fc5f367 [id=b07a3851b8cd427ba7b48f9be771544d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.671648 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb720c09c5486995391837fc54e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.671988 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.676041 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.676263 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62895377a8ac4506a2f4e10039dbdd74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.679725 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 (DEFAULT_TABLE table=default.Table_10e42a0526054267b35b1acd7a1bc12c [id=da75815357824bd3a59216d34246edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.680009 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.683768 5843 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:03.683758 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e555a89d340448bc852f2441ca850213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e555a89d340448bc852f2441ca850213 already in progress: creating tablet
I20240507 08:12:03.683974 5843 ts_tablet_manager.cc:1367] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.000s
I20240507 08:12:03.684856 5843 raft_consensus.cc:348] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.685165 5843 raft_consensus.cc:374] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.685233 5843 raft_consensus.cc:729] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.685518 5843 consensus_queue.cc:260] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.686373 5843 ts_tablet_manager.cc:1398] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 08:12:03.686758 5843 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.688405 5843 tablet_bootstrap.cc:654] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.690430 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.690780 5843 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.690910 5843 ts_tablet_manager.cc:1367] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240507 08:12:03.691673 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 8ed8bac124cb4de983f0fdbaf728975f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.691598 5843 raft_consensus.cc:348] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.691771 5843 raft_consensus.cc:374] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.691829 5843 raft_consensus.cc:729] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.691917 6547 tablet_replica.cc:330] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.692095 6547 raft_consensus.cc:2229] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.692042 5843 consensus_queue.cc:260] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.692229 6547 raft_consensus.cc:2258] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.692447 5843 ts_tablet_manager.cc:1398] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240507 08:12:03.692675 5843 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.692710 6547 ts_tablet_manager.cc:1875] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.694013 5843 tablet_bootstrap.cc:654] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.700140 6547 ts_tablet_manager.cc:1888] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.700336 6547 log.cc:1196] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/8ed8bac124cb4de983f0fdbaf728975f
I20240507 08:12:03.700752 6547 ts_tablet_manager.cc:1909] T 8ed8bac124cb4de983f0fdbaf728975f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:03.704373 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 191adef34efa4c3999d4642a80177294 (table default.Table_cc2b1c5d509b43afa9523de30d68a4d9 [id=4a0c4e6ef9464330947cf3863387b58a]) successfully deleted
W20240507 08:12:03.704737 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 62f3b6d22f7840e7b43da7a52c25a810 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f3b6d22f7840e7b43da7a52c25a810 already in progress: creating tablet
W20240507 08:12:03.705116 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 14929efb0ed049d6b554598a5b8cb494 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14929efb0ed049d6b554598a5b8cb494 already in progress: creating tablet
W20240507 08:12:03.705418 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 234e582b211940aea3471e5accb9d016 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 234e582b211940aea3471e5accb9d016 already in progress: creating tablet
W20240507 08:12:03.705729 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 767905763ba14cb8bbc52151ae9f17e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 767905763ba14cb8bbc52151ae9f17e6 already in progress: creating tablet
I20240507 08:12:03.705937 5839 tablet_bootstrap.cc:492] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:12:03.706104 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e298f11ff9f54e48b8de435f6849f39a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e298f11ff9f54e48b8de435f6849f39a already in progress: creating tablet
I20240507 08:12:03.706132 5839 ts_tablet_manager.cc:1367] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.106s user 0.002s sys 0.006s
W20240507 08:12:03.706389 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 173a2e351dea42e9938af5edb3c5c9fe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 173a2e351dea42e9938af5edb3c5c9fe already in progress: creating tablet
W20240507 08:12:03.706655 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8615f89e816e48fc8a058e5dc466918c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8615f89e816e48fc8a058e5dc466918c already in progress: creating tablet
W20240507 08:12:03.706907 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e555a89d340448bc852f2441ca850213 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e555a89d340448bc852f2441ca850213 already in progress: creating tablet
I20240507 08:12:03.707053 5839 raft_consensus.cc:348] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.707309 5839 raft_consensus.cc:374] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.707383 5839 raft_consensus.cc:729] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:03.707273 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3fce98f472ef46589dbea6b6aefa6572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fce98f472ef46589dbea6b6aefa6572 already in progress: creating tablet
I20240507 08:12:03.707665 5839 consensus_queue.cc:260] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.708530 5839 ts_tablet_manager.cc:1398] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:03.708847 5839 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.709846 5541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 121) took 1592 ms. Trace:
I20240507 08:12:03.709972 5541 rpcz_store.cc:276] 0507 08:12:02.117382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.117548 (+ 166us) service_pool.cc:225] Handling call
0507 08:12:03.709792 (+1592244us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31616}
I20240507 08:12:03.715682 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 6f6670e99e4a4442801d42950eb7ab79 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.715991 6547 tablet_replica.cc:330] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.716012 6603 raft_consensus.cc:482] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.716210 6547 raft_consensus.cc:2229] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.716131 6603 raft_consensus.cc:504] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.716367 6547 raft_consensus.cc:2258] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.716717 6603 leader_election.cc:288] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.717005 6547 ts_tablet_manager.cc:1875] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:03.708067 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 35e34b60b78b402faf51e003c97db787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e34b60b78b402faf51e003c97db787 already in progress: creating tablet
W20240507 08:12:03.717775 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8497f8ea3c9a48d281d91c90ea305cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8497f8ea3c9a48d281d91c90ea305cb2 already in progress: creating tablet
I20240507 08:12:03.717880 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e53409fe13e4a9585dbba648a25cc07 with delete_type TABLET_DATA_DELETED (replaced by tablet b0b9f85cd78748d3afce83b64b3f823b) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:03.718091 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1037c3c0af5e464bb23a311e552fcf21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1037c3c0af5e464bb23a311e552fcf21 already in progress: creating tablet
I20240507 08:12:03.718190 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 7eaf0e2936324a9fbb2426a8026886fc with delete_type TABLET_DATA_DELETED (replaced by tablet 5471c74c33cd42e8aaf47f71f61adff7) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.718684 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 648113d9b8164cc59ba238ace8f65b39 (DEFAULT_TABLE table=default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5 [id=cbd530b6bd504bf48785968ba2ac3f75]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.719190 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 648113d9b8164cc59ba238ace8f65b39. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.719364 5520 leader_election.cc:341] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5e0fb75933874a948281bf573a1723a1
I20240507 08:12:03.723997 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d (DEFAULT_TABLE table=default.Table_4659cc72cc3040e084a516672f35f2ec [id=c0a82997be2d45cca329a564fa01150b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.724453 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32ed504c4cf443e48ab285466b9e9b5d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.732543 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3fce98f472ef46589dbea6b6aefa6572 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fce98f472ef46589dbea6b6aefa6572 already in progress: creating tablet
I20240507 08:12:03.733947 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 182124e13ca749c8a2a96d38514158b1 (DEFAULT_TABLE table=default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f [id=4e15b2c7c70f426fa80aa0c92a00e35d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.734431 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 182124e13ca749c8a2a96d38514158b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.742101 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 (DEFAULT_TABLE table=default.Table_10e42a0526054267b35b1acd7a1bc12c [id=da75815357824bd3a59216d34246edc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.742563 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.743386 6614 raft_consensus.cc:482] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.743606 6614 raft_consensus.cc:504] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.744314 6614 leader_election.cc:288] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:03.745172 6513 tablet_replica.cc:330] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:03.745352 6513 raft_consensus.cc:2229] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.745489 6513 raft_consensus.cc:2258] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:03.746709 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 35e34b60b78b402faf51e003c97db787 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35e34b60b78b402faf51e003c97db787 already in progress: creating tablet
I20240507 08:12:03.748183 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:03.751348 5563 tablet_service.cc:1467] Processing CreateTablet for tablet bf8fe5157dd9491cac7e0237c3d39ec1 (DEFAULT_TABLE table=default.Table_a48b5e1d22504c999c0df606f5092147 [id=786a00558b0f42cd9700cd3200dfe1ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.751863 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf8fe5157dd9491cac7e0237c3d39ec1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.755983 6513 ts_tablet_manager.cc:1875] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.770354 5559 tablet_service.cc:1467] Processing CreateTablet for tablet ddcb720c09c5486995391837fc54e7ce (DEFAULT_TABLE table=default.Table_bcd651b469bc460c8df842333fc5f367 [id=b07a3851b8cd427ba7b48f9be771544d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:03.770530 5378 leader_election.cc:341] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.770661 5378 leader_election.cc:302] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.770761 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddcb720c09c5486995391837fc54e7ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:03.770902 6614 raft_consensus.cc:2726] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.770957 5517 leader_election.cc:341] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 191adef34efa4c3999d4642a80177294
I20240507 08:12:03.771064 5517 leader_election.cc:302] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:03.771241 5517 raft_consensus.cc:2670] T 191adef34efa4c3999d4642a80177294 P fe673739d8084e7da5d7edd722e857d8: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:03.771473 5517 leader_election.cc:341] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:12:03.771561 5517 leader_election.cc:302] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.771759 6603 raft_consensus.cc:2726] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.771872 5517 leader_election.cc:341] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5e53409fe13e4a9585dbba648a25cc07
I20240507 08:12:03.771961 5517 leader_election.cc:302] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.772153 6603 raft_consensus.cc:2726] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.772807 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.777151 6513 ts_tablet_manager.cc:1888] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:03.777244 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0fb75933874a948281bf573a1723a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.777915 6513 log.cc:1196] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8615f89e816e48fc8a058e5dc466918c
I20240507 08:12:03.778961 5839 tablet_bootstrap.cc:654] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.779106 6513 ts_tablet_manager.cc:1909] T 8615f89e816e48fc8a058e5dc466918c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:03.781111 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 8ed8bac124cb4de983f0fdbaf728975f (table default.Table_71f59cb7918c42d29419261d5f361830 [id=2e71cf49719f4d9fb07c289e1428080c]) successfully deleted
I20240507 08:12:03.781744 5840 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:03.781896 5840 ts_tablet_manager.cc:1367] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.331s user 0.005s sys 0.017s
I20240507 08:12:03.782351 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8615f89e816e48fc8a058e5dc466918c (table default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]) successfully deleted
I20240507 08:12:03.782574 5840 raft_consensus.cc:348] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.782754 5840 raft_consensus.cc:374] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.782805 5840 raft_consensus.cc:729] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.783028 5840 consensus_queue.cc:260] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.783566 5840 ts_tablet_manager.cc:1398] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:03.783797 5840 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:03.782141 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 (DEFAULT_TABLE table=default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:03.784601 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62895377a8ac4506a2f4e10039dbdd74. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:03.784991 5377 leader_election.cc:341] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:03.785676 5378 leader_election.cc:341] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.785777 5378 leader_election.cc:302] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.786116 6614 raft_consensus.cc:2726] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.786453 5517 leader_election.cc:341] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5e0fb75933874a948281bf573a1723a1
I20240507 08:12:03.786577 5517 leader_election.cc:302] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.786855 6603 raft_consensus.cc:2726] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.787572 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1037c3c0af5e464bb23a311e552fcf21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1037c3c0af5e464bb23a311e552fcf21 already in progress: creating tablet
W20240507 08:12:03.788535 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8497f8ea3c9a48d281d91c90ea305cb2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8497f8ea3c9a48d281d91c90ea305cb2 already in progress: creating tablet
I20240507 08:12:03.789443 6547 ts_tablet_manager.cc:1888] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.789635 6513 tablet_replica.cc:330] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:03.790232 6513 raft_consensus.cc:2229] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.790843 6513 raft_consensus.cc:2258] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.791801 6513 ts_tablet_manager.cc:1875] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.792547 6633 tablet.cc:1722] T b2de528d9fc046f183f70388cffab6c8 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:03.793650 6547 log.cc:1196] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/6f6670e99e4a4442801d42950eb7ab79
I20240507 08:12:03.794045 6547 ts_tablet_manager.cc:1909] T 6f6670e99e4a4442801d42950eb7ab79 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:03.794898 6547 tablet_replica.cc:330] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.795065 6547 raft_consensus.cc:2229] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.798676 6547 raft_consensus.cc:2258] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.799252 6547 ts_tablet_manager.cc:1875] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.802011 5839 tablet_bootstrap.cc:492] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:03.802166 5839 ts_tablet_manager.cc:1367] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.001s
I20240507 08:12:03.802996 5839 raft_consensus.cc:348] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.803212 5839 raft_consensus.cc:374] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.803308 5839 raft_consensus.cc:729] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.803551 5839 consensus_queue.cc:260] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.804204 5839 ts_tablet_manager.cc:1398] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:03.805070 5840 tablet_bootstrap.cc:654] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.806166 5839 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.807750 5839 tablet_bootstrap.cc:654] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.809876 5843 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.810148 5843 ts_tablet_manager.cc:1367] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.118s user 0.024s sys 0.053s
I20240507 08:12:03.811079 5843 raft_consensus.cc:348] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.811391 5843 raft_consensus.cc:374] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.811501 5843 raft_consensus.cc:729] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.811771 5843 consensus_queue.cc:260] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.812644 5843 ts_tablet_manager.cc:1398] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:03.813104 5843 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.790509 5699 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55948 (request call id 651) took 1032 ms. Trace:
I20240507 08:12:03.816066 5699 rpcz_store.cc:276] 0507 08:12:02.757747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:02.757800 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:03.158392 (+400592us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:03.158417 (+ 25us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:03.505320 (+346903us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:03.790357 (+285037us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:03.790491 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:03.818154 5843 tablet_bootstrap.cc:654] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.822283 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 28e5a380f2784e3a806afcbb5cd2519a
I20240507 08:12:03.823129 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.823460 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.823843 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
W20240507 08:12:03.824255 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 28e5a380f2784e3a806afcbb5cd2519a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28e5a380f2784e3a806afcbb5cd2519a already in progress: creating tablet
I20240507 08:12:03.825841 6547 ts_tablet_manager.cc:1888] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.825860 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e941e3377db04a6388ea05ca49010874
I20240507 08:12:03.826015 6547 log.cc:1196] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5e53409fe13e4a9585dbba648a25cc07
I20240507 08:12:03.826654 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b3fefd616dbe4d5282b1e4325f179875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:03.826861 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet e941e3377db04a6388ea05ca49010874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.827178 6547 ts_tablet_manager.cc:1909] T 5e53409fe13e4a9585dbba648a25cc07 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:03.827266 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3fefd616dbe4d5282b1e4325f179875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:03.828264 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 114380us
W20240507 08:12:03.828815 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3535f1fbfe34f6e95e9895771ae1350 has no consensus state
I20240507 08:12:03.829217 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.829843 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 consensus state has no leader
I20240507 08:12:03.830246 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8a185e645f9d40f79750397df21fbad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.830832 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a185e645f9d40f79750397df21fbad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.832542 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16b1817c74e5443c9c83a953e4514b13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.833002 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b1817c74e5443c9c83a953e4514b13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.833483 6547 tablet_replica.cc:330] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.833670 6547 raft_consensus.cc:2229] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.833788 6547 raft_consensus.cc:2258] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.834239 6547 ts_tablet_manager.cc:1875] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.834447 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 3ccc3e67eeb34c42b36edde7bacd23a5 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:03.835731 5839 tablet_bootstrap.cc:492] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:03.835835 5839 ts_tablet_manager.cc:1367] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.002s
I20240507 08:12:03.836951 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:03.836871 5839 raft_consensus.cc:348] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.837059 5839 raft_consensus.cc:374] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.837110 5839 raft_consensus.cc:729] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.837967 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 2612c023740644cb9564c44f7bfe8b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.838389 6638 tablet.cc:1722] T b2de528d9fc046f183f70388cffab6c8 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:03.838488 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:03.839392 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet e2b27de4d2d84a2f8c177a231c985785 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:03.839828 6639 tablet.cc:1722] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:03.841081 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a6cef6772a0f432891b63dd745d57873
I20240507 08:12:03.841797 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cef6772a0f432891b63dd745d57873 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.842464 5840 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:03.842581 5840 ts_tablet_manager.cc:1367] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240507 08:12:03.843454 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet e941e3377db04a6388ea05ca49010874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.843349 5840 raft_consensus.cc:348] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.843658 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 2612c023740644cb9564c44f7bfe8b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.843672 5840 raft_consensus.cc:374] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.843760 5840 raft_consensus.cc:729] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.844022 5841 tablet_replica.cc:330] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:03.844183 5841 raft_consensus.cc:2229] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.844306 5841 raft_consensus.cc:2258] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.844375 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2d4d839694e44fbdba0344b3d11b5555" } new_table_name: "default.Table_86003022cb914d9ead81ce912a28d1cd" modify_external_catalogs: true
I20240507 08:12:03.844739 5841 ts_tablet_manager.cc:1875] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:03.845660 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e941e3377db04a6388ea05ca49010874 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e941e3377db04a6388ea05ca49010874 already in progress: creating tablet
W20240507 08:12:03.846220 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.847836 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cef6772a0f432891b63dd745d57873 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.848651 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 41f75cf9f00b4d03b72101f1a0668df3
I20240507 08:12:03.849370 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_544538e6b4974f38b63b2e1f3214ba25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:03.849614 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:03.849874 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_544538e6b4974f38b63b2e1f3214ba25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.853699 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:03.854348 5843 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.854568 5843 ts_tablet_manager.cc:1367] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240507 08:12:03.837271 5839 consensus_queue.cc:260] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.855288 5843 raft_consensus.cc:348] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.856789 5843 raft_consensus.cc:374] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.856897 5843 raft_consensus.cc:729] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.856828 5840 consensus_queue.cc:260] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.857169 5843 consensus_queue.cc:260] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.857474 5839 ts_tablet_manager.cc:1398] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240507 08:12:03.858220 5839 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.858235 5843 ts_tablet_manager.cc:1398] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:03.859179 5843 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.859776 5839 tablet_bootstrap.cc:654] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.861106 5840 ts_tablet_manager.cc:1398] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240507 08:12:03.861361 5840 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:03.861564 5841 ts_tablet_manager.cc:1888] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.861749 5841 log.cc:1196] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:03.862104 5841 ts_tablet_manager.cc:1909] T 2612c023740644cb9564c44f7bfe8b61 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:03.862776 5840 tablet_bootstrap.cc:654] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.862325 5843 tablet_bootstrap.cc:654] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.864063 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 2612c023740644cb9564c44f7bfe8b61 (table default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]) successfully deleted
W20240507 08:12:03.864529 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a6cef6772a0f432891b63dd745d57873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6cef6772a0f432891b63dd745d57873 already in progress: creating tablet
W20240507 08:12:03.864791 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 41f75cf9f00b4d03b72101f1a0668df3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f75cf9f00b4d03b72101f1a0668df3 already in progress: creating tablet
I20240507 08:12:03.871039 6638 tablet.cc:1722] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:03.871111 6547 ts_tablet_manager.cc:1888] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.871367 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_61d23e54efcc415c9f29270b3e622bc0" } new_table_name: "default.Table_7c4f79308328475f8aa4076850acb6ec" modify_external_catalogs: true
I20240507 08:12:03.874439 6547 log.cc:1196] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/7eaf0e2936324a9fbb2426a8026886fc
I20240507 08:12:03.875226 6547 ts_tablet_manager.cc:1909] T 7eaf0e2936324a9fbb2426a8026886fc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:03.877034 5843 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.877200 5843 ts_tablet_manager.cc:1367] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20240507 08:12:03.877910 5843 raft_consensus.cc:348] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.878154 5843 raft_consensus.cc:374] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.878237 5843 raft_consensus.cc:729] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.878485 5843 consensus_queue.cc:260] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.878963 5843 ts_tablet_manager.cc:1398] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:03.879235 5843 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.880599 5843 tablet_bootstrap.cc:654] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.880829 6547 tablet_replica.cc:330] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.881042 6547 raft_consensus.cc:2229] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.881556 6547 raft_consensus.cc:2258] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.882632 6614 raft_consensus.cc:482] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.882755 6614 raft_consensus.cc:504] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.883301 6614 leader_election.cc:288] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:03.884016 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:03.884529 5377 leader_election.cc:341] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.893975 6547 ts_tablet_manager.cc:1875] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:03.894307 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.894641 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet e941e3377db04a6388ea05ca49010874 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.895082 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 6f6670e99e4a4442801d42950eb7ab79 (table default.Table_fea6e34b279d43b2918afeac8c53ceec [id=d8e126be70904a50930744b45f809acc]) successfully deleted
I20240507 08:12:03.895509 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5e53409fe13e4a9585dbba648a25cc07 (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:12:03.895779 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 7eaf0e2936324a9fbb2426a8026886fc (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:12:03.901057 5839 tablet_bootstrap.cc:492] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:03.901222 5839 ts_tablet_manager.cc:1367] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.003s
I20240507 08:12:03.901926 5839 raft_consensus.cc:348] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.907047 5839 raft_consensus.cc:374] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.907167 5839 raft_consensus.cc:729] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.907438 5839 consensus_queue.cc:260] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.909112 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2612c023740644cb9564c44f7bfe8b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.912297 5839 ts_tablet_manager.cc:1398] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240507 08:12:03.913986 5839 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:03.921223 5843 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.921437 5843 ts_tablet_manager.cc:1367] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.011s
I20240507 08:12:03.922806 5843 raft_consensus.cc:348] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.923061 5843 raft_consensus.cc:374] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.923158 5843 raft_consensus.cc:729] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.923694 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e2b27de4d2d84a2f8c177a231c985785
I20240507 08:12:03.924247 5259 catalog_manager.cc:6554] ReplaceTablet: tablet e2b27de4d2d84a2f8c177a231c985785 deleted and replaced by tablet db92da02e5af49f5a13a814f3ce4f9c3
I20240507 08:12:03.924651 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_de8250b1e1a9404f9a3c984b62bd309b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.925263 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de8250b1e1a9404f9a3c984b62bd309b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.926527 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b27de4d2d84a2f8c177a231c985785 with delete_type TABLET_DATA_DELETED (replaced by tablet db92da02e5af49f5a13a814f3ce4f9c3) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:03.927248 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e2b27de4d2d84a2f8c177a231c985785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2b27de4d2d84a2f8c177a231c985785 already in progress: creating tablet
I20240507 08:12:03.927935 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9735652cdc0d447080df766780402da6" } new_table_name: "default.Table_48b51787d9ce4ff1b9b8744ed86885bb" modify_external_catalogs: true
I20240507 08:12:03.927269 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_17ed4a4959964b3e859909cccb93a18f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.931013 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17ed4a4959964b3e859909cccb93a18f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.923422 5843 consensus_queue.cc:260] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.932582 5843 ts_tablet_manager.cc:1398] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240507 08:12:03.932900 5843 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.934607 5843 tablet_bootstrap.cc:654] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.936062 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b27de4d2d84a2f8c177a231c985785 with delete_type TABLET_DATA_DELETED (replaced by tablet db92da02e5af49f5a13a814f3ce4f9c3) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:03.939013 6547 ts_tablet_manager.cc:1888] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:03.939139 6547 log.cc:1196] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5e0fb75933874a948281bf573a1723a1
W20240507 08:12:03.941552 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b0542325c80241fca7e317cda64302a9 consensus state has no leader
I20240507 08:12:03.943562 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:03.944202 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:03.951539 5843 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.951681 5843 ts_tablet_manager.cc:1367] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.003s
I20240507 08:12:03.951715 5840 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:03.951984 5840 ts_tablet_manager.cc:1367] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.091s user 0.006s sys 0.003s
I20240507 08:12:03.952373 5843 raft_consensus.cc:348] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.952582 5843 raft_consensus.cc:374] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.952637 5843 raft_consensus.cc:729] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.952798 6547 ts_tablet_manager.cc:1909] T 5e0fb75933874a948281bf573a1723a1 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:03.952729 5840 raft_consensus.cc:348] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.952903 5840 raft_consensus.cc:374] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.952956 5840 raft_consensus.cc:729] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.952879 5843 consensus_queue.cc:260] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.953358 5843 ts_tablet_manager.cc:1398] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:03.954067 5843 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.954177 6603 raft_consensus.cc:482] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.954298 6603 raft_consensus.cc:504] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.955588 6603 leader_election.cc:288] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.956261 5840 consensus_queue.cc:260] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.958302 5843 tablet_bootstrap.cc:654] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.958611 5840 ts_tablet_manager.cc:1398] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240507 08:12:03.958884 5840 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:03.960433 5840 tablet_bootstrap.cc:654] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.960762 5843 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.960880 5843 ts_tablet_manager.cc:1367] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
W20240507 08:12:03.961055 5520 leader_election.cc:341] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: bde62374dc7f4a67a071b6b658e501c4
I20240507 08:12:03.961506 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 8615f89e816e48fc8a058e5dc466918c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.961733 6547 tablet_replica.cc:330] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:03.961880 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet a6cef6772a0f432891b63dd745d57873 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.961905 6547 raft_consensus.cc:2229] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:03.962055 6547 raft_consensus.cc:2258] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:03.962028 5843 raft_consensus.cc:348] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.962179 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 41f75cf9f00b4d03b72101f1a0668df3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.962203 5843 raft_consensus.cc:374] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.962253 5843 raft_consensus.cc:729] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.962460 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet c84a2018f1c442e2a4dfc1c6b9c5ceef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.962483 5843 consensus_queue.cc:260] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.962967 5843 ts_tablet_manager.cc:1398] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:03.963254 5843 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.964030 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet e2b27de4d2d84a2f8c177a231c985785 with delete_type TABLET_DATA_DELETED (replaced by tablet db92da02e5af49f5a13a814f3ce4f9c3) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:03.964675 5843 tablet_bootstrap.cc:654] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.964936 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:03.966190 6603 raft_consensus.cc:482] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.966462 5839 tablet_bootstrap.cc:654] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.966352 6603 raft_consensus.cc:504] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.967144 6603 leader_election.cc:288] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.967552 6547 ts_tablet_manager.cc:1875] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:03.967833 5520 leader_election.cc:341] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: c84a2018f1c442e2a4dfc1c6b9c5ceef
W20240507 08:12:03.969528 5378 leader_election.cc:341] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.969671 5378 leader_election.cc:302] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.969957 6614 raft_consensus.cc:2726] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.970574 5517 leader_election.cc:341] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: bde62374dc7f4a67a071b6b658e501c4
I20240507 08:12:03.970708 5517 leader_election.cc:302] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.970937 6603 raft_consensus.cc:2726] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:03.971557 5517 leader_election.cc:341] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:12:03.971712 5517 leader_election.cc:302] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:03.971987 6603 raft_consensus.cc:2726] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:03.973232 5843 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.973354 5843 ts_tablet_manager.cc:1367] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240507 08:12:03.973881 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5e0fb75933874a948281bf573a1723a1 (table default.Table_80a52d858eaf480cab22d552d7c8eff5 [id=c4316da76e1b4a57b230f785839847eb]) successfully deleted
I20240507 08:12:03.974159 5843 raft_consensus.cc:348] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.974350 5843 raft_consensus.cc:374] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.974406 5843 raft_consensus.cc:729] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:03.974282 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 28e5a380f2784e3a806afcbb5cd2519a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28e5a380f2784e3a806afcbb5cd2519a already in progress: creating tablet
I20240507 08:12:03.974604 5843 consensus_queue.cc:260] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:03.974867 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e941e3377db04a6388ea05ca49010874 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e941e3377db04a6388ea05ca49010874 already in progress: creating tablet
W20240507 08:12:03.975178 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2612c023740644cb9564c44f7bfe8b61 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2612c023740644cb9564c44f7bfe8b61 already in progress: creating tablet
I20240507 08:12:03.975193 5843 ts_tablet_manager.cc:1398] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:03.975517 5843 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.977048 5843 tablet_bootstrap.cc:654] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.985282 5843 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:03.985479 5843 ts_tablet_manager.cc:1367] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20240507 08:12:03.985731 6553 raft_consensus.cc:482] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:03.985893 6553 raft_consensus.cc:504] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.986212 5843 raft_consensus.cc:348] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.986369 5843 raft_consensus.cc:374] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.986424 5843 raft_consensus.cc:729] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.986610 5843 consensus_queue.cc:260] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.986719 6553 leader_election.cc:288] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:03.987134 5843 ts_tablet_manager.cc:1398] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:03.987392 5843 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:03.987285 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:03.987941 5654 leader_election.cc:341] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:03.988219 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:03.988400 5443 raft_consensus.cc:2454] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:03.989562 5843 tablet_bootstrap.cc:654] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.989511 5656 leader_election.cc:302] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:03.989917 6553 raft_consensus.cc:2781] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:03.989990 6553 raft_consensus.cc:482] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:03.990043 6553 raft_consensus.cc:3037] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:03.993290 5840 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:03.993418 5840 ts_tablet_manager.cc:1367] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.001s
I20240507 08:12:03.994153 5840 raft_consensus.cc:348] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.994325 5840 raft_consensus.cc:374] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:03.994377 5840 raft_consensus.cc:729] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:03.994585 5840 consensus_queue.cc:260] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:03.994719 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 63ccdad494794192b3c37c8d5cbfcb17 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:03.994949 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 3a065a4441f449a299515dda078f2d24 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:03.995558 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd6933ac8ee3452db6edf3c255247b33 consensus state has no leader
I20240507 08:12:03.996860 5840 ts_tablet_manager.cc:1398] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:12:03.996706 5264 catalog_manager.cc:5484] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:03.997123 5840 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:03.997189 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:03.998675 5840 tablet_bootstrap.cc:654] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:03.999881 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 996b698544b34df2a849c0fee48c315f (DEFAULT_TABLE table=default.Table_b17843ca835c4017a73c4bf3baf9afea [id=561099724c6a433ca70640bcbf446a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.000283 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996b698544b34df2a849c0fee48c315f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.000619 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 (DEFAULT_TABLE table=default.Table_3dc5823424924e0bbff545ee187770d8 [id=f5777fc347f1459c84728e103a384554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.000773 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 5782cfa5eca74c19a2e724a9124597f2 (table default.Table_dc3c2099e75b478ba03e7084ecd2e317 [id=c6a13f9902b0424ca8df2184779a407c]). Expected version 2 got 0
I20240507 08:12:04.000918 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c68ddcfa7c40c180b6b12fabe165d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.000988 5258 catalog_manager.cc:5484] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:04.001567 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:04.005486 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet bde62374dc7f4a67a071b6b658e501c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.005843 5417 tablet_service.cc:1467] Processing CreateTablet for tablet db3ad8ba1c3a4eb98b59aed4eca73115 (DEFAULT_TABLE table=default.Table_9774c61bef464f96ab38fafb81f92ca2 [id=596dc026024742ca8b12f99b3f1e7d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.005904 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 996b698544b34df2a849c0fee48c315f (DEFAULT_TABLE table=default.Table_b17843ca835c4017a73c4bf3baf9afea [id=561099724c6a433ca70640bcbf446a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.006186 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3ad8ba1c3a4eb98b59aed4eca73115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.006321 5550 tablet_service.cc:1467] Processing CreateTablet for tablet db3ad8ba1c3a4eb98b59aed4eca73115 (DEFAULT_TABLE table=default.Table_9774c61bef464f96ab38fafb81f92ca2 [id=596dc026024742ca8b12f99b3f1e7d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.006750 5559 tablet_service.cc:1467] Processing CreateTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 (DEFAULT_TABLE table=default.Table_6d4810aabea24b539d53b0e43c7298ff [id=9bd0e351458948dcb8bcf819c2004eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.005514 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 (DEFAULT_TABLE table=default.Table_3dc5823424924e0bbff545ee187770d8 [id=f5777fc347f1459c84728e103a384554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.012385 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 996b698544b34df2a849c0fee48c315f (DEFAULT_TABLE table=default.Table_b17843ca835c4017a73c4bf3baf9afea [id=561099724c6a433ca70640bcbf446a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.013176 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 (DEFAULT_TABLE table=default.Table_3dc5823424924e0bbff545ee187770d8 [id=f5777fc347f1459c84728e103a384554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.013631 5691 tablet_service.cc:1467] Processing CreateTablet for tablet db3ad8ba1c3a4eb98b59aed4eca73115 (DEFAULT_TABLE table=default.Table_9774c61bef464f96ab38fafb81f92ca2 [id=596dc026024742ca8b12f99b3f1e7d9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.013983 5684 tablet_service.cc:1467] Processing CreateTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 (DEFAULT_TABLE table=default.Table_6d4810aabea24b539d53b0e43c7298ff [id=9bd0e351458948dcb8bcf819c2004eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.014149 5551 tablet_service.cc:1467] Processing CreateTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 (DEFAULT_TABLE table=default.Table_b2afebeaafe84471bf8b1ae310b46036 [id=8980b428efe040fbbeb01cee44aa565f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.014341 5688 tablet_service.cc:1467] Processing CreateTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 (DEFAULT_TABLE table=default.Table_b2afebeaafe84471bf8b1ae310b46036 [id=8980b428efe040fbbeb01cee44aa565f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.017349 5410 tablet_service.cc:1467] Processing CreateTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 (DEFAULT_TABLE table=default.Table_6d4810aabea24b539d53b0e43c7298ff [id=9bd0e351458948dcb8bcf819c2004eef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.017704 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a570e39a52d84d3dbcaeb9ea005d9f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.018460 5839 tablet_bootstrap.cc:492] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.018592 5839 ts_tablet_manager.cc:1367] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.000s
I20240507 08:12:04.019243 5839 raft_consensus.cc:348] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.019430 5839 raft_consensus.cc:374] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.019475 5839 raft_consensus.cc:729] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.019737 5839 consensus_queue.cc:260] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.020607 5839 ts_tablet_manager.cc:1398] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:04.021721 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 (DEFAULT_TABLE table=default.Table_b2afebeaafe84471bf8b1ae310b46036 [id=8980b428efe040fbbeb01cee44aa565f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.022044 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3535f1fbfe34f6e95e9895771ae1350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.024644 5840 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.024827 5840 ts_tablet_manager.cc:1367] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240507 08:12:04.024914 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_17ed4a4959964b3e859909cccb93a18f" } modify_external_catalogs: true
I20240507 08:12:04.025138 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_17ed4a4959964b3e859909cccb93a18f" } modify_external_catalogs: true
I20240507 08:12:04.025637 5840 raft_consensus.cc:348] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.025837 5840 raft_consensus.cc:374] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.025897 5840 raft_consensus.cc:729] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.026167 5840 consensus_queue.cc:260] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.027596 5840 ts_tablet_manager.cc:1398] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:12:04.027973 6553 raft_consensus.cc:504] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.028506 5840 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.028584 6553 leader_election.cc:288] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.029106 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996b698544b34df2a849c0fee48c315f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.029335 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3535f1fbfe34f6e95e9895771ae1350. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.029525 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a570e39a52d84d3dbcaeb9ea005d9f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.029455 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:04.029718 5443 raft_consensus.cc:3037] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.029855 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3ad8ba1c3a4eb98b59aed4eca73115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.030005 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c68ddcfa7c40c180b6b12fabe165d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.032214 5443 raft_consensus.cc:2454] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:04.035576 6513 ts_tablet_manager.cc:1888] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.035760 6513 log.cc:1196] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c33710c6b55744c4ba0843c228abded3
I20240507 08:12:04.035614 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.036659 6513 ts_tablet_manager.cc:1909] T c33710c6b55744c4ba0843c228abded3 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:04.036844 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a570e39a52d84d3dbcaeb9ea005d9f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.037119 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:04.037348 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a3535f1fbfe34f6e95e9895771ae1350. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:04.037657 5654 leader_election.cc:341] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.037783 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db3ad8ba1c3a4eb98b59aed4eca73115. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.037843 5839 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.040519 5839 tablet_bootstrap.cc:654] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.041342 5840 tablet_bootstrap.cc:654] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.042388 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eec3f4ed076b4741aa5573b9805a83ce" } modify_external_catalogs: true
I20240507 08:12:04.042632 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_eec3f4ed076b4741aa5573b9805a83ce" } modify_external_catalogs: true
I20240507 08:12:04.044260 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30fd0c57577c474e932f08b9d7725680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.044802 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30fd0c57577c474e932f08b9d7725680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.045814 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4737c51589da4300bf37cc2638784041" } new_table_name: "default.Table_b19f64f84d434d8684f3b784794f2f52" modify_external_catalogs: true
I20240507 08:12:04.046785 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d08d2f517e6420aa76bcf5ec7928959" } modify_external_catalogs: true
I20240507 08:12:04.047008 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93c68ddcfa7c40c180b6b12fabe165d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.046999 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f10e0f30c1f14fa18ead66c614f8473e" } new_table_name: "default.Table_73aa505d7d4a4389bbd269ee956f5d8a" modify_external_catalogs: true
I20240507 08:12:04.047621 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 996b698544b34df2a849c0fee48c315f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.048431 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e61891ba7bda49f99a67145edfe93e5b" } modify_external_catalogs: true
I20240507 08:12:04.048486 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6709887aaee84c1ba633b7f910c797e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.048605 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e61891ba7bda49f99a67145edfe93e5b" } modify_external_catalogs: true
W20240507 08:12:04.048952 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6709887aaee84c1ba633b7f910c797e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.046998 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d08d2f517e6420aa76bcf5ec7928959" } modify_external_catalogs: true
I20240507 08:12:04.054104 5656 leader_election.cc:302] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.057412 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3ccc3e67eeb34c42b36edde7bacd23a5
I20240507 08:12:04.057992 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 3ccc3e67eeb34c42b36edde7bacd23a5 deleted and replaced by tablet f89efb8414dc49369d361adf908927e6
I20240507 08:12:04.058296 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 with delete_type TABLET_DATA_DELETED (replaced by tablet f89efb8414dc49369d361adf908927e6) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.059402 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 with delete_type TABLET_DATA_DELETED (replaced by tablet f89efb8414dc49369d361adf908927e6) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.060845 6614 raft_consensus.cc:482] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:04.061002 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ccc3e67eeb34c42b36edde7bacd23a5 already in progress: creating tablet
I20240507 08:12:04.060981 6614 raft_consensus.cc:504] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.061676 6614 leader_election.cc:288] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.062055 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.062322 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.062525 5378 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.062726 6553 raft_consensus.cc:2781] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:04.062934 5377 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.062952 6553 raft_consensus.cc:686] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:04.063028 5377 leader_election.cc:302] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.063202 6553 consensus_queue.cc:237] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.063478 6614 raft_consensus.cc:2726] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.064966 5257 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 96c770050462474faae57c5a5d41b0b8
I20240507 08:12:04.065330 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_483c951eac554e4c9a5a5f6e16e2b109" } modify_external_catalogs: true
I20240507 08:12:04.065512 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_483c951eac554e4c9a5a5f6e16e2b109" } modify_external_catalogs: true
I20240507 08:12:04.073261 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.073480 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.073709 5841 tablet_replica.cc:330] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:04.073892 5841 raft_consensus.cc:2229] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.074041 5841 raft_consensus.cc:2258] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.074590 5841 ts_tablet_manager.cc:1875] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.078568 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 with delete_type TABLET_DATA_DELETED (replaced by tablet f89efb8414dc49369d361adf908927e6) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.078651 5585 raft_consensus.cc:3037] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.093981 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_42ec5a6050fd445ebe2a0545cc26cbc0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.094682 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42ec5a6050fd445ebe2a0545cc26cbc0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.096323 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bc4c56d06c6c4c9f83495859be8a5c5e
I20240507 08:12:04.096946 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet b27f89f6db6746fb870dad2a3fb27457 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:04.098145 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2bb648fe98e74248b5cd65603de58a02 consensus state has no leader
W20240507 08:12:04.098611 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec08c0d8a1ab444489340415303a5739 consensus state has no leader
I20240507 08:12:04.098980 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d8a297a72b474e508184e1f32a2a37cf
I20240507 08:12:04.099566 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.099762 6508 raft_consensus.cc:482] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.099874 6508 raft_consensus.cc:504] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.100504 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.100567 6508 leader_election.cc:288] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.100791 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a297a72b474e508184e1f32a2a37cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.101472 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22334378eec4ab3885d0f9d2d812a3e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.101928 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a297a72b474e508184e1f32a2a37cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.102340 6614 raft_consensus.cc:482] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.102509 6614 raft_consensus.cc:504] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.103299 6614 leader_election.cc:288] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:04.103863 5656 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:12:04.104529 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.105201 5378 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.105652 5654 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.105784 5654 leader_election.cc:302] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.106040 6508 raft_consensus.cc:2726] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.106540 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.107602 5377 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.107753 5377 leader_election.cc:302] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.107899 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet d8a297a72b474e508184e1f32a2a37cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.108107 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet bc4c56d06c6c4c9f83495859be8a5c5e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.108731 6614 raft_consensus.cc:2726] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.115482 6603 raft_consensus.cc:482] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.115681 6603 raft_consensus.cc:504] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.116475 6603 leader_election.cc:288] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.117766 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.117440 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4306751d16444ec5948c2ba5831063f4" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.118810 5520 leader_election.cc:302] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:04.119367 6603 raft_consensus.cc:2726] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.119383 5839 tablet_bootstrap.cc:492] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.123178 5839 ts_tablet_manager.cc:1367] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20240507 08:12:04.124749 5839 raft_consensus.cc:348] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.124997 5839 raft_consensus.cc:374] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.125080 5839 raft_consensus.cc:729] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.125458 5839 consensus_queue.cc:260] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.126878 5839 ts_tablet_manager.cc:1398] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:04.128089 5839 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.129735 5839 tablet_bootstrap.cc:654] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.130355 6513 tablet_replica.cc:330] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:04.130575 6513 raft_consensus.cc:2229] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.130934 6513 raft_consensus.cc:2258] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.131561 6513 ts_tablet_manager.cc:1875] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.133499 6547 ts_tablet_manager.cc:1888] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.133755 6547 log.cc:1196] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/8615f89e816e48fc8a058e5dc466918c
I20240507 08:12:04.134244 6547 ts_tablet_manager.cc:1909] T 8615f89e816e48fc8a058e5dc466918c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:04.138106 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_91b52ae3fbb140a7bad7e02bd6e58c18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.138720 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91b52ae3fbb140a7bad7e02bd6e58c18 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.140635 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c33710c6b55744c4ba0843c228abded3 (table default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]) successfully deleted
I20240507 08:12:04.141482 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c7be75b22b2d41878a28edd9a44b6456"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.142031 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7be75b22b2d41878a28edd9a44b6456 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.142242 5843 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.142398 5843 ts_tablet_manager.cc:1367] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.155s user 0.027s sys 0.033s
I20240507 08:12:04.143553 5843 raft_consensus.cc:348] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.143797 5843 raft_consensus.cc:374] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.143846 5843 raft_consensus.cc:729] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.144093 5843 consensus_queue.cc:260] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.147401 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:04.148561 6632 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:04.150614 6633 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:04.153499 5843 ts_tablet_manager.cc:1398] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240507 08:12:04.153879 5843 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.154372 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7f18168519594979b02d651f093b2115
I20240507 08:12:04.155786 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5643e8bc0add4df885536ff947d357c6
I20240507 08:12:04.156805 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5643e8bc0add4df885536ff947d357c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.156967 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5643e8bc0add4df885536ff947d357c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.157735 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 5643e8bc0add4df885536ff947d357c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.157932 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f18168519594979b02d651f093b2115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.158121 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f18168519594979b02d651f093b2115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.158857 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f18168519594979b02d651f093b2115 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:04.158794 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 136577us
I20240507 08:12:04.170773 6553 raft_consensus.cc:482] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.170971 6553 raft_consensus.cc:504] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.171774 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_44802e7664e8479c97e888d0a2b4dbf4" } new_table_name: "default.Table_afe14a8a082943dd9b05964cec987540" modify_external_catalogs: true
I20240507 08:12:04.172163 6553 leader_election.cc:288] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.172283 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089db17ca904a04878d83209001c54e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.172469 5443 raft_consensus.cc:2454] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:04.172889 5656 leader_election.cc:302] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:04.173911 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:04.175071 6553 raft_consensus.cc:2781] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.175187 6553 raft_consensus.cc:482] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.175279 6553 raft_consensus.cc:3037] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.178654 5839 tablet_bootstrap.cc:492] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.178820 5839 ts_tablet_manager.cc:1367] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.004s
I20240507 08:12:04.179852 5839 raft_consensus.cc:348] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.180178 5839 raft_consensus.cc:374] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.180259 5839 raft_consensus.cc:729] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.180920 5843 tablet_bootstrap.cc:654] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.182741 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089db17ca904a04878d83209001c54e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.183315 5654 leader_election.cc:341] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.180533 5839 consensus_queue.cc:260] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.185290 6547 tablet_replica.cc:330] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:04.185461 6547 raft_consensus.cc:2229] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.185639 6547 raft_consensus.cc:2258] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.186316 5839 ts_tablet_manager.cc:1398] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20240507 08:12:04.186601 5839 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.187624 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 8615f89e816e48fc8a058e5dc466918c (table default.Table_4557db659d1e4591a2bffcc5077e92dd [id=2aa73b17e4d0463e82585acb4616ba43]) successfully deleted
W20240507 08:12:04.187882 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a6cef6772a0f432891b63dd745d57873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6cef6772a0f432891b63dd745d57873 already in progress: creating tablet
W20240507 08:12:04.188149 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 41f75cf9f00b4d03b72101f1a0668df3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f75cf9f00b4d03b72101f1a0668df3 already in progress: creating tablet
I20240507 08:12:04.188750 5839 tablet_bootstrap.cc:654] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.191795 6547 ts_tablet_manager.cc:1875] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.196837 5843 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.197082 5843 ts_tablet_manager.cc:1367] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.002s
I20240507 08:12:04.198135 5843 raft_consensus.cc:348] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.198428 5843 raft_consensus.cc:374] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.198501 5843 raft_consensus.cc:729] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.198802 5843 consensus_queue.cc:260] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.199872 5843 ts_tablet_manager.cc:1398] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:12:04.203810 5843 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.206848 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b27f89f6db6746fb870dad2a3fb27457
I20240507 08:12:04.207365 5262 catalog_manager.cc:6554] ReplaceTablet: tablet b27f89f6db6746fb870dad2a3fb27457 deleted and replaced by tablet e3783fc7fe434696898ba14ab77a9058
I20240507 08:12:04.208461 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b27f89f6db6746fb870dad2a3fb27457 with delete_type TABLET_DATA_DELETED (replaced by tablet e3783fc7fe434696898ba14ab77a9058) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.209244 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet b27f89f6db6746fb870dad2a3fb27457 with delete_type TABLET_DATA_DELETED (replaced by tablet e3783fc7fe434696898ba14ab77a9058) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.210317 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet b27f89f6db6746fb870dad2a3fb27457 with delete_type TABLET_DATA_DELETED (replaced by tablet e3783fc7fe434696898ba14ab77a9058) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.211102 6663 tablet.cc:1722] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:04.211771 6497 raft_consensus.cc:482] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:04.212303 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbfbcafc930c496ca7c8f008c934e26d has no consensus state
I20240507 08:12:04.213279 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5c3aae31efac4a47bc0d47090bd103c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.213858 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c3aae31efac4a47bc0d47090bd103c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.215214 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_04756d6d06eb47d2a3c61debf3d6eef1" } new_table_name: "default.Table_881b71c36b4048b5acf1ab063ccbf33b" modify_external_catalogs: true
I20240507 08:12:04.217996 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b8da67c846954c4a95c5a20759624d6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.218469 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8da67c846954c4a95c5a20759624d6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.219712 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a77bc176ba0948c796f99b08bcc16d1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.220247 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a77bc176ba0948c796f99b08bcc16d1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.221661 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c083a659194342cdbdbb42898c701c6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.222424 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dc3c2099e75b478ba03e7084ecd2e317" } new_table_name: "default.Table_ee5279fc19a046aebae1f622389f59a2" modify_external_catalogs: true
I20240507 08:12:04.224330 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_73bd4c93636247cb834bbf7b46c39ad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.225185 5654 consensus_peers.cc:485] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:04.225732 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c083a659194342cdbdbb42898c701c6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.226348 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73bd4c93636247cb834bbf7b46c39ad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.227978 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e48a2b4a0e4088ad009a7f8fe77f3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.211907 6497 raft_consensus.cc:504] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.236267 6497 leader_election.cc:288] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.236940 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.237252 5841 ts_tablet_manager.cc:1888] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.237362 5841 log.cc:1196] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/a7df45c39d1b490db97c72a0eb6cf3ba
W20240507 08:12:04.237715 5378 leader_election.cc:341] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.238009 5843 tablet_bootstrap.cc:654] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.238121 6553 raft_consensus.cc:504] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.238797 6553 leader_election.cc:288] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:04.240195 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124ede6467314270a2d99e0e45097c52 consensus state has no leader
I20240507 08:12:04.242790 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089db17ca904a04878d83209001c54e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:04.242833 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.242974 5443 raft_consensus.cc:3037] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.245132 5841 ts_tablet_manager.cc:1909] T a7df45c39d1b490db97c72a0eb6cf3ba P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:04.245355 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5089db17ca904a04878d83209001c54e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:04.245841 5654 leader_election.cc:341] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.245896 5377 leader_election.cc:341] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.245994 5377 leader_election.cc:302] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.246351 6497 raft_consensus.cc:2726] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.247510 5839 tablet_bootstrap.cc:492] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.247673 5839 ts_tablet_manager.cc:1367] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.002s
I20240507 08:12:04.248513 5839 raft_consensus.cc:348] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.248740 5839 raft_consensus.cc:374] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.248798 5839 raft_consensus.cc:729] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:04.250654 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5782cfa5eca74c19a2e724a9124597f2 consensus state has no leader
I20240507 08:12:04.249006 5839 consensus_queue.cc:260] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.258940 5839 ts_tablet_manager.cc:1398] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20240507 08:12:04.259074 5840 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.259828 5839 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.260509 5843 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.260648 5843 ts_tablet_manager.cc:1367] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.000s
I20240507 08:12:04.261456 5843 raft_consensus.cc:348] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.261703 5843 raft_consensus.cc:374] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.261762 5843 raft_consensus.cc:729] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.261967 5843 consensus_queue.cc:260] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.262523 5843 ts_tablet_manager.cc:1398] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:04.264819 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 072702ef2a6d43248132da6aa3a2d493 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.259240 5840 ts_tablet_manager.cc:1367] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.231s user 0.038s sys 0.051s
I20240507 08:12:04.265031 5839 tablet_bootstrap.cc:654] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.268656 5840 raft_consensus.cc:348] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.268882 5840 raft_consensus.cc:374] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.268941 5840 raft_consensus.cc:729] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.269153 5840 consensus_queue.cc:260] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.269729 5840 ts_tablet_manager.cc:1398] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:04.270005 5840 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.271035 6553 raft_consensus.cc:482] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.271245 6553 raft_consensus.cc:504] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.271751 5840 tablet_bootstrap.cc:654] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.271891 6553 leader_election.cc:288] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.274500 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a2c072fbeb35480fa276c46cbc1c6aa7" } modify_external_catalogs: true
I20240507 08:12:04.275517 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a2c072fbeb35480fa276c46cbc1c6aa7" } modify_external_catalogs: true
I20240507 08:12:04.275907 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cd02a49359743c4b4e540ab0338d8ed" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.276546 5654 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.278090 5656 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:04.278254 5656 leader_election.cc:302] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.278654 6553 raft_consensus.cc:2726] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.279349 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_48b51787d9ce4ff1b9b8744ed86885bb [id=0a1bc06a351549d5aea075ad8d026476]). Expected version 4 got 3
I20240507 08:12:04.279400 5843 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.279755 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 9ced879f35e14106b2609c78b7f24002 (table default.Table_dea0e2b1b090454cbd15a00f7e792d6e [id=aa09d786696945d5b4060819737a6e11]). Expected version 1 got 0
I20240507 08:12:04.279969 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
I20240507 08:12:04.280499 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 4716c1285e91432a92b25ab13ada2779 (table default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb [id=bd83f9b19311401a8ff3fd39e8c75e8f]). Expected version 1 got 0
I20240507 08:12:04.281008 5843 tablet_bootstrap.cc:654] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.281219 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:04.282081 5260 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a] alter complete (version 2)
I20240507 08:12:04.290557 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 807660cbb3ae417c94f410a92251dcf9 (DEFAULT_TABLE table=default.Table_544538e6b4974f38b63b2e1f3214ba25 [id=8ec296b587af4db9bc74c5eb7297de9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.294036 5260 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3] alter complete (version 1)
I20240507 08:12:04.295598 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.295867 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.295969 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.296195 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.296476 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.302479 5260 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:04.304134 5260 catalog_manager.cc:5484] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:04.303206 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1462e4ac39014fb0aafe7485fe690e29" } modify_external_catalogs: true
I20240507 08:12:04.304811 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1462e4ac39014fb0aafe7485fe690e29" } modify_external_catalogs: true
I20240507 08:12:04.303480 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8f7c06393ac54a15842f97319969eff1" } new_table_name: "default.Table_64eb52ba1a3746b692ef0a3512c3eea1" modify_external_catalogs: true
I20240507 08:12:04.303741 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d8b43861f60b4b6189e6386d097240a9" } new_table_name: "default.Table_a63ec02af6ab4e85830582d44267f8c1" modify_external_catalogs: true
I20240507 08:12:04.302896 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_96d7f5313ab8422bbfd790cea475d2ac" } modify_external_catalogs: true
I20240507 08:12:04.308861 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_96d7f5313ab8422bbfd790cea475d2ac" } modify_external_catalogs: true
I20240507 08:12:04.309896 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 85dc58133d2b4f5496b4e570ae6494c0 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:04.311933 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7f5a117d33fe42ae985e25ef355c1def"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.312659 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f5a117d33fe42ae985e25ef355c1def in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.317871 6553 consensus_queue.cc:1035] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:04.326893 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.328344 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 807660cbb3ae417c94f410a92251dcf9 (DEFAULT_TABLE table=default.Table_544538e6b4974f38b63b2e1f3214ba25 [id=8ec296b587af4db9bc74c5eb7297de9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.330642 5840 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.330770 5843 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.330850 5840 ts_tablet_manager.cc:1367] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.012s
I20240507 08:12:04.330910 5843 ts_tablet_manager.cc:1367] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.002s
I20240507 08:12:04.331596 5843 raft_consensus.cc:348] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.331615 5840 raft_consensus.cc:348] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.331802 5843 raft_consensus.cc:374] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.331837 5840 raft_consensus.cc:374] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.331859 5843 raft_consensus.cc:729] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.331905 5840 raft_consensus.cc:729] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.332080 5843 consensus_queue.cc:260] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.332347 5841 tablet_replica.cc:330] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:04.332212 5840 consensus_queue.cc:260] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.333853 5843 ts_tablet_manager.cc:1398] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:04.334112 5843 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.334784 6513 ts_tablet_manager.cc:1888] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.334903 6513 log.cc:1196] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/107d9b3f95e14044854ad0bf91a268ee
I20240507 08:12:04.335268 6513 ts_tablet_manager.cc:1909] T 107d9b3f95e14044854ad0bf91a268ee P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:04.335566 5843 tablet_bootstrap.cc:654] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.337152 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 07a9324570984372af7b9574f0cf455a
I20240507 08:12:04.337755 5443 raft_consensus.cc:2454] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:04.338023 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a9324570984372af7b9574f0cf455a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.339229 5840 ts_tablet_manager.cc:1398] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:12:04.339295 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a9324570984372af7b9574f0cf455a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.339596 5840 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.341593 5841 raft_consensus.cc:2229] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.341985 5841 raft_consensus.cc:2258] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.342413 5840 tablet_bootstrap.cc:654] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.342509 5841 ts_tablet_manager.cc:1875] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.343753 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:04.344583 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.344929 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 85dc58133d2b4f5496b4e570ae6494c0 deleted and replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009
W20240507 08:12:04.345747 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bcefbe3a60f84cadb5fd3e69c6efdd02 consensus state has no leader
W20240507 08:12:04.346539 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13fa1a807d3548d6836652379d455b0d consensus state has no leader
I20240507 08:12:04.347246 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8e1f91718260466d99acfc9892dea536
I20240507 08:12:04.348294 5843 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.348429 5843 ts_tablet_manager.cc:1367] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20240507 08:12:04.349125 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 039dcaf121b94ebab94ec522346d9d82
I20240507 08:12:04.349423 5843 raft_consensus.cc:348] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.349640 5843 raft_consensus.cc:374] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.349700 5843 raft_consensus.cc:729] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.349939 5843 consensus_queue.cc:260] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.350551 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.351043 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1f91718260466d99acfc9892dea536 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.351475 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 039dcaf121b94ebab94ec522346d9d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.353839 5843 ts_tablet_manager.cc:1398] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:04.354133 5843 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.355722 6497 raft_consensus.cc:482] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.355852 6497 raft_consensus.cc:504] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.356539 6497 leader_election.cc:288] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.357148 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807660cbb3ae417c94f410a92251dcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.357787 5843 tablet_bootstrap.cc:654] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.358896 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.359437 5378 leader_election.cc:341] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.360363 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3634efacf6494a4eabe5a4d9451e64a6" } modify_external_catalogs: true
I20240507 08:12:04.360569 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3634efacf6494a4eabe5a4d9451e64a6" } modify_external_catalogs: true
I20240507 08:12:04.361282 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 039dcaf121b94ebab94ec522346d9d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.361524 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.361932 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.362005 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ced552bcfdd14427a0b6990736d3165c" } modify_external_catalogs: true
I20240507 08:12:04.362166 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ced552bcfdd14427a0b6990736d3165c" } modify_external_catalogs: true
I20240507 08:12:04.362298 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1f91718260466d99acfc9892dea536 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.367506 5656 leader_election.cc:302] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.367975 6553 raft_consensus.cc:2781] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:04.368113 6553 raft_consensus.cc:686] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:04.368396 6553 consensus_queue.cc:237] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.368980 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 53bc9bf1c04e4015a152cb750184c9ce from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:04.370051 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet a7df45c39d1b490db97c72a0eb6cf3ba (table default.Table_0fafe65af5f34be4b333cb6baa327f94 [id=58282ffddcba4e4db7736a325a4a530a]) successfully deleted
I20240507 08:12:04.370733 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 07a9324570984372af7b9574f0cf455a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.370919 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1f91718260466d99acfc9892dea536 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.370955 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.371122 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 039dcaf121b94ebab94ec522346d9d82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:04.372303 5377 leader_election.cc:341] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.372406 5377 leader_election.cc:302] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.373075 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.373275 6497 raft_consensus.cc:2726] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.376917 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b5d4993c287a40cf964877b5464372a6
I20240507 08:12:04.378399 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4993c287a40cf964877b5464372a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.381333 6508 raft_consensus.cc:482] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.381520 6508 raft_consensus.cc:504] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.382198 6508 leader_election.cc:288] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.382162 5551 tablet_service.cc:1467] Processing CreateTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 (DEFAULT_TABLE table=default.Table_16b1817c74e5443c9c83a953e4514b13 [id=41171a0db55148579e8ef1c8478a2f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.382746 6614 raft_consensus.cc:482] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.382867 6614 raft_consensus.cc:504] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.383383 6614 leader_election.cc:288] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.383488 6614 raft_consensus.cc:482] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.383589 6614 raft_consensus.cc:504] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.383921 5549 tablet_service.cc:1467] Processing CreateTablet for tablet a903c17e01c241bcb13787ef551395d2 (DEFAULT_TABLE table=default.Table_8a185e645f9d40f79750397df21fbad6 [id=1eae5ce04b1c49f9b865453999452df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.384090 6614 leader_election.cc:288] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.384667 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa14901cdf14b72bea365b04ad521bd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.384840 5443 raft_consensus.cc:2454] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:04.384915 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.385099 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b19d8e28f164c588946ea5e81ec320e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.385450 5378 leader_election.cc:341] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.385648 5378 leader_election.cc:341] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.385931 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b19d8e28f164c588946ea5e81ec320e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.386153 5716 raft_consensus.cc:2454] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:04.386346 5692 tablet_service.cc:1467] Processing CreateTablet for tablet a903c17e01c241bcb13787ef551395d2 (DEFAULT_TABLE table=default.Table_8a185e645f9d40f79750397df21fbad6 [id=1eae5ce04b1c49f9b865453999452df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.386417 5691 tablet_service.cc:1467] Processing CreateTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 (DEFAULT_TABLE table=default.Table_16b1817c74e5443c9c83a953e4514b13 [id=41171a0db55148579e8ef1c8478a2f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.386788 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a903c17e01c241bcb13787ef551395d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.386850 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.387193 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4993c287a40cf964877b5464372a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.387674 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab350a3f4f3341c2ad0dda3ce054e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.386809 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b81673c2cce64edba3bbf1a45e6c3e49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.390239 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b81673c2cce64edba3bbf1a45e6c3e49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.391420 6497 raft_consensus.cc:482] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.391554 6497 raft_consensus.cc:504] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.392133 6497 leader_election.cc:288] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.392468 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.392691 5656 leader_election.cc:302] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:04.392903 5378 leader_election.cc:341] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.392956 6508 raft_consensus.cc:2781] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.393038 6508 raft_consensus.cc:482] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.393121 6508 raft_consensus.cc:3037] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.388547 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 3b16a81588394d28945331e233e1f1db from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:04.389467 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 53bc9bf1c04e4015a152cb750184c9ce
I20240507 08:12:04.394035 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 53bc9bf1c04e4015a152cb750184c9ce deleted and replaced by tablet f416d08748ac483ab0ce98240e9eb036
I20240507 08:12:04.391963 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0c48c9720c1a4a8b849730f28f6e00d9
I20240507 08:12:04.396620 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.396775 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.396828 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.396972 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.397361 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.397325 5411 tablet_service.cc:1467] Processing CreateTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 (DEFAULT_TABLE table=default.Table_16b1817c74e5443c9c83a953e4514b13 [id=41171a0db55148579e8ef1c8478a2f4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.397500 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.397626 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.397325 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 807660cbb3ae417c94f410a92251dcf9 (DEFAULT_TABLE table=default.Table_544538e6b4974f38b63b2e1f3214ba25 [id=8ec296b587af4db9bc74c5eb7297de9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.397716 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.397758 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab350a3f4f3341c2ad0dda3ce054e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.397900 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.397984 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807660cbb3ae417c94f410a92251dcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.398051 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.398196 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b5d4993c287a40cf964877b5464372a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.398712 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa14901cdf14b72bea365b04ad521bd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.400055 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a903c17e01c241bcb13787ef551395d2 (DEFAULT_TABLE table=default.Table_8a185e645f9d40f79750397df21fbad6 [id=1eae5ce04b1c49f9b865453999452df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.400368 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a903c17e01c241bcb13787ef551395d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.401760 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a903c17e01c241bcb13787ef551395d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.404062 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 807660cbb3ae417c94f410a92251dcf9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:04.407088 5654 consensus_peers.cc:485] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:04.407253 5654 leader_election.cc:341] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.407874 5377 leader_election.cc:302] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:04.409107 5377 leader_election.cc:341] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.409204 5377 leader_election.cc:302] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.409482 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab350a3f4f3341c2ad0dda3ce054e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.412444 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5e958b4da1dd43ec9cc03a7554860b1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.412613 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c48c9720c1a4a8b849730f28f6e00d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.413017 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:04.413195 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e958b4da1dd43ec9cc03a7554860b1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.413232 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.413694 6614 raft_consensus.cc:2781] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.413781 6614 raft_consensus.cc:482] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.413847 6614 raft_consensus.cc:3037] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.416591 6497 raft_consensus.cc:2726] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:04.416858 5377 leader_election.cc:341] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.416949 5377 leader_election.cc:302] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.417408 6497 raft_consensus.cc:2726] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.418745 6513 tablet_replica.cc:330] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:04.418916 6513 raft_consensus.cc:2229] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:04.419147 6513 pending_rounds.cc:62] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:04.419189 6513 pending_rounds.cc:68] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924767663767552 op_type: NO_OP noop_request { }
I20240507 08:12:04.419283 6513 raft_consensus.cc:2866] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:04.419348 6513 raft_consensus.cc:2258] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.420348 5843 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.420465 5843 ts_tablet_manager.cc:1367] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.006s
I20240507 08:12:04.421175 5843 raft_consensus.cc:348] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.421409 5843 raft_consensus.cc:374] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.421468 5843 raft_consensus.cc:729] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.421690 6497 raft_consensus.cc:482] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.421741 5843 consensus_queue.cc:260] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.421805 6497 raft_consensus.cc:504] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.422813 6497 leader_election.cc:288] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.426060 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 107d9b3f95e14044854ad0bf91a268ee (table default.Table_351291351f3c4972ac5f9cd896d68312 [id=ccea3f017cc84cce95f3010023a76bb4]) successfully deleted
W20240507 08:12:04.426319 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 28e5a380f2784e3a806afcbb5cd2519a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28e5a380f2784e3a806afcbb5cd2519a already in progress: creating tablet
I20240507 08:12:04.423249 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.426576 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e941e3377db04a6388ea05ca49010874 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e941e3377db04a6388ea05ca49010874 already in progress: creating tablet
W20240507 08:12:04.427050 5378 leader_election.cc:341] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.429109 6513 ts_tablet_manager.cc:1875] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.429548 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.430029 5377 leader_election.cc:341] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.430130 5377 leader_election.cc:302] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.430446 6497 raft_consensus.cc:2726] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.430563 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6a2e33773a684a20903120d0e1c2e4c3" } new_table_name: "default.Table_e359119a48d246408f844aca603ee8c0" modify_external_catalogs: true
I20240507 08:12:04.431614 6547 ts_tablet_manager.cc:1888] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.431711 6547 log.cc:1196] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c84a2018f1c442e2a4dfc1c6b9c5ceef
I20240507 08:12:04.431996 6547 ts_tablet_manager.cc:1909] T c84a2018f1c442e2a4dfc1c6b9c5ceef P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:04.432298 6508 raft_consensus.cc:504] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.433148 6508 leader_election.cc:288] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.434109 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa14901cdf14b72bea365b04ad521bd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:04.434284 5443 raft_consensus.cc:3037] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.434917 6614 raft_consensus.cc:504] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.438055 6614 leader_election.cc:288] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.438140 6547 tablet_replica.cc:330] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:04.438299 6547 raft_consensus.cc:2229] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.438376 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b19d8e28f164c588946ea5e81ec320e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:04.438820 5378 leader_election.cc:341] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.439046 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c84a2018f1c442e2a4dfc1c6b9c5ceef (table default.Table_b1add7221be940138ee5b9e2e796e92c [id=70d5dddc40a3453bb64b8019c2a18046]) successfully deleted
W20240507 08:12:04.439345 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e2b27de4d2d84a2f8c177a231c985785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2b27de4d2d84a2f8c177a231c985785 already in progress: creating tablet
I20240507 08:12:04.438441 6547 raft_consensus.cc:2258] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.443327 6547 ts_tablet_manager.cc:1875] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.446170 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfa14901cdf14b72bea365b04ad521bd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:04.447654 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b19d8e28f164c588946ea5e81ec320e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:04.447832 5720 raft_consensus.cc:3037] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.448753 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8a36aab1e10b41c589b8b0eac87af0db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.449337 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a36aab1e10b41c589b8b0eac87af0db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.450371 5654 leader_election.cc:341] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.450987 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e292ef7310c5489f8b3c60c61ab75b92" } new_table_name: "default.Table_e07a25a325a64352a0433eee53f843ab" modify_external_catalogs: true
I20240507 08:12:04.453071 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_479f84438e6548f1806133a52949d796" } new_table_name: "default.Table_848f877aafbe4022bb2a1ec887d8245b" modify_external_catalogs: true
I20240507 08:12:04.452726 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9ab22bf322414792b4044b2f77b6144a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.454799 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ab22bf322414792b4044b2f77b6144a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.455951 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3b48d6ff43424a76a86e1880f32c59f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.456507 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b48d6ff43424a76a86e1880f32c59f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.470948 5839 tablet_bootstrap.cc:492] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.471082 5839 ts_tablet_manager.cc:1367] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.212s user 0.012s sys 0.008s
I20240507 08:12:04.471799 5839 raft_consensus.cc:348] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.471997 5839 raft_consensus.cc:374] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.472091 5839 raft_consensus.cc:729] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.472338 5839 consensus_queue.cc:260] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.474442 5840 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.474612 5840 ts_tablet_manager.cc:1367] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.135s user 0.008s sys 0.012s
I20240507 08:12:04.475281 5840 raft_consensus.cc:348] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.475461 5840 raft_consensus.cc:374] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.475517 5840 raft_consensus.cc:729] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.475745 5840 consensus_queue.cc:260] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.476338 5840 ts_tablet_manager.cc:1398] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:04.476622 5840 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.479485 5840 tablet_bootstrap.cc:654] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.489902 5843 ts_tablet_manager.cc:1398] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.069s user 0.004s sys 0.004s
I20240507 08:12:04.491358 5699 tablet_service.cc:1467] Processing CreateTablet for tablet fbfbcafc930c496ca7c8f008c934e26d (DEFAULT_TABLE table=default.Table_afe14a8a082943dd9b05964cec987540 [id=7696378debc0437c99191689e156614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.491791 5688 tablet_service.cc:1467] Processing CreateTablet for tablet a2606b9f58d64d588581e45611ba11b2 (DEFAULT_TABLE table=default.Table_b5eee064314f4431b951651d05ed1e13 [id=55800abeb08b453c978e757e1bd46167]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.492687 5843 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.492656 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 7564834150854a7abd1fe3932c7e9e4b (DEFAULT_TABLE table=default.Table_601dd7f065e343eba100d1a64f0a72a0 [id=1184f2a4b87b49a0adacfbf3c14598ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.493135 5414 tablet_service.cc:1467] Processing CreateTablet for tablet fbfbcafc930c496ca7c8f008c934e26d (DEFAULT_TABLE table=default.Table_afe14a8a082943dd9b05964cec987540 [id=7696378debc0437c99191689e156614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.493230 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 7564834150854a7abd1fe3932c7e9e4b (DEFAULT_TABLE table=default.Table_601dd7f065e343eba100d1a64f0a72a0 [id=1184f2a4b87b49a0adacfbf3c14598ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.493290 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a2606b9f58d64d588581e45611ba11b2 (DEFAULT_TABLE table=default.Table_b5eee064314f4431b951651d05ed1e13 [id=55800abeb08b453c978e757e1bd46167]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.494045 5693 tablet_service.cc:1467] Processing CreateTablet for tablet de7cddb2c4f743f7ae02fd92061a418c (DEFAULT_TABLE table=default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2 [id=f7f693983a8d42f487789d90cebaa872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.494693 5843 tablet_bootstrap.cc:654] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.495249 5840 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.495414 5840 ts_tablet_manager.cc:1367] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.009s
I20240507 08:12:04.496176 5840 raft_consensus.cc:348] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.496412 5840 raft_consensus.cc:374] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.496492 5840 raft_consensus.cc:729] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.496722 5840 consensus_queue.cc:260] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.497025 6553 raft_consensus.cc:482] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.497125 6553 raft_consensus.cc:504] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.498226 6553 leader_election.cc:288] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.498903 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.472843 5839 ts_tablet_manager.cc:1398] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20240507 08:12:04.499505 5656 leader_election.cc:341] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.499642 5839 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.498127 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.500566 5654 leader_election.cc:341] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.500671 5654 leader_election.cc:302] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.501286 6553 raft_consensus.cc:2726] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.501341 5839 tablet_bootstrap.cc:654] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.501874 5840 ts_tablet_manager.cc:1398] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:04.502238 5840 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.507896 5843 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.508637 5843 ts_tablet_manager.cc:1367] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20240507 08:12:04.509897 5843 raft_consensus.cc:348] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.512590 5840 tablet_bootstrap.cc:654] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.513260 5843 raft_consensus.cc:374] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.513381 5843 raft_consensus.cc:729] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.513672 5843 consensus_queue.cc:260] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.513998 6614 raft_consensus.cc:482] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.514226 5843 ts_tablet_manager.cc:1398] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:04.514503 5843 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.514784 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 42809e2f45e9466594377b41ae920f3b (DEFAULT_TABLE table=default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a [id=0aee284b067849a88dd1d8b9104adb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.516069 5843 tablet_bootstrap.cc:654] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.517395 5563 tablet_service.cc:1467] Processing CreateTablet for tablet de7cddb2c4f743f7ae02fd92061a418c (DEFAULT_TABLE table=default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2 [id=f7f693983a8d42f487789d90cebaa872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.517480 5550 tablet_service.cc:1467] Processing CreateTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.517778 5559 tablet_service.cc:1467] Processing CreateTablet for tablet fbfbcafc930c496ca7c8f008c934e26d (DEFAULT_TABLE table=default.Table_afe14a8a082943dd9b05964cec987540 [id=7696378debc0437c99191689e156614d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.517843 5697 tablet_service.cc:1467] Processing CreateTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.518230 5551 tablet_service.cc:1467] Processing CreateTablet for tablet a2606b9f58d64d588581e45611ba11b2 (DEFAULT_TABLE table=default.Table_b5eee064314f4431b951651d05ed1e13 [id=55800abeb08b453c978e757e1bd46167]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.518308 5425 tablet_service.cc:1467] Processing CreateTablet for tablet de7cddb2c4f743f7ae02fd92061a418c (DEFAULT_TABLE table=default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2 [id=f7f693983a8d42f487789d90cebaa872]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.518612 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 7564834150854a7abd1fe3932c7e9e4b (DEFAULT_TABLE table=default.Table_601dd7f065e343eba100d1a64f0a72a0 [id=1184f2a4b87b49a0adacfbf3c14598ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.518791 5426 tablet_service.cc:1467] Processing CreateTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.519014 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 42809e2f45e9466594377b41ae920f3b (DEFAULT_TABLE table=default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a [id=0aee284b067849a88dd1d8b9104adb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.519207 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 42809e2f45e9466594377b41ae920f3b (DEFAULT_TABLE table=default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a [id=0aee284b067849a88dd1d8b9104adb68]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.530459 5583 raft_consensus.cc:3037] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.530455 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3b16a81588394d28945331e233e1f1db
I20240507 08:12:04.531076 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 3b16a81588394d28945331e233e1f1db deleted and replaced by tablet 5f9846e0eab74388af231e9942a6881a
I20240507 08:12:04.532706 5413 tablet_service.cc:1467] Processing CreateTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.533084 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b16a81588394d28945331e233e1f1db with delete_type TABLET_DATA_DELETED (replaced by tablet 5f9846e0eab74388af231e9942a6881a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.533416 5443 raft_consensus.cc:2454] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:04.533439 5686 tablet_service.cc:1467] Processing CreateTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.533524 5555 tablet_service.cc:1467] Processing CreateTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 (DEFAULT_TABLE table=default.Table_f4a5ad85280b49289eeef7eea3c2849d [id=963bb8f67cbd418b9778b49d3b6eaacb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.533876 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2606b9f58d64d588581e45611ba11b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.533897 5841 ts_tablet_manager.cc:1888] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.533975 5841 log.cc:1196] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5089db17ca904a04878d83209001c54e
I20240507 08:12:04.534118 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7564834150854a7abd1fe3932c7e9e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.534281 5841 ts_tablet_manager.cc:1909] T 5089db17ca904a04878d83209001c54e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:04.534513 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db92da02e5af49f5a13a814f3ce4f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.534595 5720 raft_consensus.cc:2454] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:04.535053 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42809e2f45e9466594377b41ae920f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.535451 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db92da02e5af49f5a13a814f3ce4f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.535589 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacfe980b2aa4a60ad05bbe102a94266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.535787 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b16a81588394d28945331e233e1f1db with delete_type TABLET_DATA_DELETED (replaced by tablet 5f9846e0eab74388af231e9942a6881a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.535908 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2606b9f58d64d588581e45611ba11b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.536041 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfbcafc930c496ca7c8f008c934e26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.536255 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db92da02e5af49f5a13a814f3ce4f9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.536603 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42809e2f45e9466594377b41ae920f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.537406 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfbcafc930c496ca7c8f008c934e26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.537775 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b16a81588394d28945331e233e1f1db with delete_type TABLET_DATA_DELETED (replaced by tablet 5f9846e0eab74388af231e9942a6881a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.537827 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2606b9f58d64d588581e45611ba11b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.537783 5547 tablet_service.cc:1467] Processing CreateTablet for tablet be1b36ca5b6041d780e851f137d3a4c9 (DEFAULT_TABLE table=default.Table_de8250b1e1a9404f9a3c984b62bd309b [id=6325b34aec0b45d981c7a0825903409a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.538229 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7564834150854a7abd1fe3932c7e9e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.538619 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7cddb2c4f743f7ae02fd92061a418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.539346 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.539577 5422 tablet_service.cc:1467] Processing CreateTablet for tablet be1b36ca5b6041d780e851f137d3a4c9 (DEFAULT_TABLE table=default.Table_de8250b1e1a9404f9a3c984b62bd309b [id=6325b34aec0b45d981c7a0825903409a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.540889 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.540983 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e215266f54e27a76f813f517b7ac5 (DEFAULT_TABLE table=default.Table_b3fefd616dbe4d5282b1e4325f179875 [id=b3871737014340d19fbcfd939d7ec59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.541707 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e215266f54e27a76f813f517b7ac5 (DEFAULT_TABLE table=default.Table_b3fefd616dbe4d5282b1e4325f179875 [id=b3871737014340d19fbcfd939d7ec59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.541718 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 0d7e215266f54e27a76f813f517b7ac5 (DEFAULT_TABLE table=default.Table_b3fefd616dbe4d5282b1e4325f179875 [id=b3871737014340d19fbcfd939d7ec59a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.542131 5656 leader_election.cc:302] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.543381 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.543884 5377 leader_election.cc:302] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.544381 6497 raft_consensus.cc:482] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.544512 6497 raft_consensus.cc:504] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.545135 6497 leader_election.cc:288] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.545241 6497 raft_consensus.cc:482] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.545328 6497 raft_consensus.cc:504] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.545841 6497 leader_election.cc:288] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.545962 6497 raft_consensus.cc:2781] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:04.546051 6497 raft_consensus.cc:686] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:04.546263 6497 consensus_queue.cc:237] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.547282 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7564834150854a7abd1fe3932c7e9e4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.547806 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42809e2f45e9466594377b41ae920f3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.548280 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacfe980b2aa4a60ad05bbe102a94266. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.548776 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cacfe980b2aa4a60ad05bbe102a94266. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:04.549695 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet fbfbcafc930c496ca7c8f008c934e26d (table default.Table_afe14a8a082943dd9b05964cec987540 [id=7696378debc0437c99191689e156614d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet fbfbcafc930c496ca7c8f008c934e26d consensus state has no leader
I20240507 08:12:04.556736 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7cddb2c4f743f7ae02fd92061a418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.561091 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfbcafc930c496ca7c8f008c934e26d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.561900 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de7cddb2c4f743f7ae02fd92061a418c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.566464 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.566874 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.567670 5378 leader_election.cc:341] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.567858 5378 leader_election.cc:341] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.568332 5691 tablet_service.cc:1467] Processing CreateTablet for tablet be1b36ca5b6041d780e851f137d3a4c9 (DEFAULT_TABLE table=default.Table_de8250b1e1a9404f9a3c984b62bd309b [id=6325b34aec0b45d981c7a0825903409a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.573884 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1b36ca5b6041d780e851f137d3a4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.574384 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccbe2e368d14b56859d62d1cc742f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.576076 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1b36ca5b6041d780e851f137d3a4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.576694 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7e215266f54e27a76f813f517b7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.577206 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7e215266f54e27a76f813f517b7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.578716 5841 tablet_replica.cc:330] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:04.578883 5841 raft_consensus.cc:2229] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.579001 5841 raft_consensus.cc:2258] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.579493 5841 ts_tablet_manager.cc:1875] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.582085 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5089db17ca904a04878d83209001c54e (table default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]) successfully deleted
W20240507 08:12:04.582437 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bc4c56d06c6c4c9f83495859be8a5c5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4c56d06c6c4c9f83495859be8a5c5e already in progress: creating tablet
W20240507 08:12:04.582685 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d8a297a72b474e508184e1f32a2a37cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8a297a72b474e508184e1f32a2a37cf already in progress: creating tablet
W20240507 08:12:04.582895 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5643e8bc0add4df885536ff947d357c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5643e8bc0add4df885536ff947d357c6 already in progress: creating tablet
W20240507 08:12:04.583112 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7f18168519594979b02d651f093b2115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f18168519594979b02d651f093b2115 already in progress: creating tablet
W20240507 08:12:04.583319 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b27f89f6db6746fb870dad2a3fb27457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27f89f6db6746fb870dad2a3fb27457 already in progress: creating tablet
I20240507 08:12:04.584275 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccbe2e368d14b56859d62d1cc742f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.589749 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_48b51787d9ce4ff1b9b8744ed86885bb [id=0a1bc06a351549d5aea075ad8d026476] alter complete (version 4)
I20240507 08:12:04.590317 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_50c935f08e4f4f4caeb2643cf499aace"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.590991 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50c935f08e4f4f4caeb2643cf499aace in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.592510 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e57fcc9d421a4abdb75cb707e06febbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.592818 6675 raft_consensus.cc:2781] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:04.592978 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e57fcc9d421a4abdb75cb707e06febbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.592973 6675 raft_consensus.cc:686] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:04.593279 6675 consensus_queue.cc:237] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:04.594493 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aa485207d0f140c99a9386cb4002c480 consensus state has no leader
I20240507 08:12:04.594918 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2472cc2477394ac9addcebfd33bea4aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.595346 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2472cc2477394ac9addcebfd33bea4aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.596442 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2e44c3f6f96e417b92878d8ba4ff88a7 consensus state has no leader
I20240507 08:12:04.596649 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 3a065a4441f449a299515dda078f2d24 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:04.597203 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3da110b2f7014fdca83ebeeb4c70c3d1" } modify_external_catalogs: true
I20240507 08:12:04.597385 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3da110b2f7014fdca83ebeeb4c70c3d1" } modify_external_catalogs: true
W20240507 08:12:04.597885 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0ddb7dd28f43488581d3a75bf1d68a8a consensus state has no leader
I20240507 08:12:04.598269 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4c3d879d3da3408ea46e6ae4ab672eca" } new_table_name: "default.Table_d563809a6ab04242827f32b2b0a6a40f" modify_external_catalogs: true
I20240507 08:12:04.589437 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_299103db24834157bce006563e9b48a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.602056 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_299103db24834157bce006563e9b48a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.603272 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_26e7316a42a14b8fae7baa93114ebcd2" } new_table_name: "default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc" modify_external_catalogs: true
I20240507 08:12:04.606887 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:04.609787 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.609858 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.610270 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb" } modify_external_catalogs: true
I20240507 08:12:04.610563 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb" } modify_external_catalogs: true
I20240507 08:12:04.612010 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.612236 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.626866 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8dd482935ec64031b96ef89534429ac3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.627560 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dd482935ec64031b96ef89534429ac3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.630750 6513 ts_tablet_manager.cc:1888] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:04.630898 6513 log.cc:1196] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:04.631270 6513 ts_tablet_manager.cc:1909] T 2612c023740644cb9564c44f7bfe8b61 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:04.635080 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 2612c023740644cb9564c44f7bfe8b61 (table default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]) successfully deleted
W20240507 08:12:04.635839 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a6cef6772a0f432891b63dd745d57873 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6cef6772a0f432891b63dd745d57873 already in progress: creating tablet
W20240507 08:12:04.636585 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 41f75cf9f00b4d03b72101f1a0668df3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41f75cf9f00b4d03b72101f1a0668df3 already in progress: creating tablet
W20240507 08:12:04.637166 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e2b27de4d2d84a2f8c177a231c985785 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2b27de4d2d84a2f8c177a231c985785 already in progress: creating tablet
I20240507 08:12:04.637461 6513 tablet_replica.cc:330] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:04.641238 6513 raft_consensus.cc:2229] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.641454 6513 raft_consensus.cc:2258] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.642048 6513 ts_tablet_manager.cc:1875] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.644814 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ccbe2e368d14b56859d62d1cc742f90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.645025 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d7e215266f54e27a76f813f517b7ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.645362 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be1b36ca5b6041d780e851f137d3a4c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:04.667807 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30fd0c57577c474e932f08b9d7725680" } modify_external_catalogs: true
I20240507 08:12:04.668061 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30fd0c57577c474e932f08b9d7725680" } modify_external_catalogs: true
I20240507 08:12:04.680051 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16b866688308450cb4d4f81ccafa1afb" } modify_external_catalogs: true
I20240507 08:12:04.680282 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16b866688308450cb4d4f81ccafa1afb" } modify_external_catalogs: true
I20240507 08:12:04.681764 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 33875331045146a9a02559a0b37d5c6f from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:04.677083 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 133202us
I20240507 08:12:04.682479 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.683928 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2961308b2bc546de99dfee745feced1e" } modify_external_catalogs: true
I20240507 08:12:04.684134 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2961308b2bc546de99dfee745feced1e" } modify_external_catalogs: true
I20240507 08:12:04.685751 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:04.686215 6547 ts_tablet_manager.cc:1888] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:12:04.687629 5377 leader_election.cc:341] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.687835 5377 leader_election.cc:302] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:04.688242 5377 leader_election.cc:341] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.688414 5377 leader_election.cc:302] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.688721 6676 raft_consensus.cc:482] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.688835 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:04.688839 6676 raft_consensus.cc:504] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.689780 6676 leader_election.cc:288] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.689960 6676 raft_consensus.cc:2726] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.690178 6676 raft_consensus.cc:2726] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.690806 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45d3fbbe3764261a3a24523a77c6443" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.691341 5378 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.691639 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.691845 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.704162 6547 log.cc:1196] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/bde62374dc7f4a67a071b6b658e501c4
I20240507 08:12:04.704488 5840 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.704622 5840 ts_tablet_manager.cc:1367] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.202s user 0.025s sys 0.055s
I20240507 08:12:04.705336 5840 raft_consensus.cc:348] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.705519 5840 raft_consensus.cc:374] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.705613 5840 raft_consensus.cc:729] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.705848 5840 consensus_queue.cc:260] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.706364 5840 ts_tablet_manager.cc:1398] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:04.710494 6547 ts_tablet_manager.cc:1909] T bde62374dc7f4a67a071b6b658e501c4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:04.713366 5840 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.714990 5840 tablet_bootstrap.cc:654] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.715066 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f1b5b7cea20745ecb2e767e67c4c8b38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.715754 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1b5b7cea20745ecb2e767e67c4c8b38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.716176 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45d3fbbe3764261a3a24523a77c6443" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.717185 5377 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.718307 5377 leader_election.cc:302] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.718386 5843 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.718904 5843 ts_tablet_manager.cc:1367] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.204s user 0.020s sys 0.065s
W20240507 08:12:04.719111 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2d45d93ed0d34806b9d9451c101f10af consensus state has no leader
I20240507 08:12:04.719296 6684 raft_consensus.cc:2726] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.719476 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 279e95d9804c44eaa2c23a4ec02c84ff
I20240507 08:12:04.719622 5843 raft_consensus.cc:348] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.719817 5843 raft_consensus.cc:374] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.719888 5843 raft_consensus.cc:729] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.721338 5843 consensus_queue.cc:260] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.721529 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4716c1285e91432a92b25ab13ada2779
I20240507 08:12:04.721537 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.721906 5843 ts_tablet_manager.cc:1398] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:04.722882 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 4716c1285e91432a92b25ab13ada2779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.723843 5843 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.723918 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 4716c1285e91432a92b25ab13ada2779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.724139 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.725637 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 4716c1285e91432a92b25ab13ada2779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.726353 5843 tablet_bootstrap.cc:654] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.728781 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.731557 6684 raft_consensus.cc:482] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.731765 6684 raft_consensus.cc:504] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.732832 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027b4cefa4d41659f819ee4326470c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.733323 5378 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.733793 5841 ts_tablet_manager.cc:1888] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.733930 5841 log.cc:1196] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/dfa14901cdf14b72bea365b04ad521bd
I20240507 08:12:04.734309 5841 ts_tablet_manager.cc:1909] T dfa14901cdf14b72bea365b04ad521bd P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:04.739301 6684 leader_election.cc:288] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.740037 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027b4cefa4d41659f819ee4326470c4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.740703 5377 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.740887 5377 leader_election.cc:302] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.741300 6684 raft_consensus.cc:2726] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.756207 5258 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 8e613f19e22d4c90911b4714998d9846
I20240507 08:12:04.756608 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 79524df2415146ab926425c42068626c
I20240507 08:12:04.757364 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 33875331045146a9a02559a0b37d5c6f
I20240507 08:12:04.757797 5261 catalog_manager.cc:6554] ReplaceTablet: tablet 33875331045146a9a02559a0b37d5c6f deleted and replaced by tablet 4ac55233449844d2bd40fee0be427e13
I20240507 08:12:04.759976 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 79524df2415146ab926425c42068626c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.760247 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.761118 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 279e95d9804c44eaa2c23a4ec02c84ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.761253 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet dfa14901cdf14b72bea365b04ad521bd (table default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]) successfully deleted
W20240507 08:12:04.762504 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 07a9324570984372af7b9574f0cf455a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9324570984372af7b9574f0cf455a already in progress: creating tablet
W20240507 08:12:04.763156 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8e1f91718260466d99acfc9892dea536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e1f91718260466d99acfc9892dea536 already in progress: creating tablet
I20240507 08:12:04.763504 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 79524df2415146ab926425c42068626c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.763841 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:04.763942 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 85dc58133d2b4f5496b4e570ae6494c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85dc58133d2b4f5496b4e570ae6494c0 already in progress: creating tablet
I20240507 08:12:04.764410 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:04.764539 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 039dcaf121b94ebab94ec522346d9d82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 039dcaf121b94ebab94ec522346d9d82 already in progress: creating tablet
I20240507 08:12:04.764859 5841 tablet_replica.cc:330] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:04.765076 5841 raft_consensus.cc:2229] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.765406 5841 raft_consensus.cc:2258] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.766121 5841 ts_tablet_manager.cc:1875] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.768801 5839 tablet_bootstrap.cc:492] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.769021 5839 ts_tablet_manager.cc:1367] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.269s user 0.007s sys 0.015s
I20240507 08:12:04.769037 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 63685e9ef255477db589fa572b087b57
I20240507 08:12:04.769827 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 63685e9ef255477db589fa572b087b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.770241 5839 raft_consensus.cc:348] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.770483 5839 raft_consensus.cc:374] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.770539 5839 raft_consensus.cc:729] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.770761 5839 consensus_queue.cc:260] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.771350 5839 ts_tablet_manager.cc:1398] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:04.771337 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 63685e9ef255477db589fa572b087b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.771607 5839 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.773165 5839 tablet_bootstrap.cc:654] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.777702 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 63685e9ef255477db589fa572b087b57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.782447 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 79524df2415146ab926425c42068626c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.783767 5840 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.783905 5840 ts_tablet_manager.cc:1367] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.071s user 0.013s sys 0.019s
I20240507 08:12:04.784616 5840 raft_consensus.cc:348] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.784816 5840 raft_consensus.cc:374] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.784864 5840 raft_consensus.cc:729] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.785143 5840 consensus_queue.cc:260] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.786044 5840 ts_tablet_manager.cc:1398] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:04.786758 5840 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.789634 5843 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.789810 5843 ts_tablet_manager.cc:1367] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.066s user 0.018s sys 0.008s
I20240507 08:12:04.789932 6675 raft_consensus.cc:482] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.790063 6675 raft_consensus.cc:504] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.790829 6675 leader_election.cc:288] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.791620 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.792063 5443 raft_consensus.cc:2454] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:04.792484 5656 leader_election.cc:302] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:04.793385 6632 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:04.794826 5840 tablet_bootstrap.cc:654] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.795818 6663 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:04.796325 5843 raft_consensus.cc:348] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.796784 6675 raft_consensus.cc:2781] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.796898 6675 raft_consensus.cc:482] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.796988 6675 raft_consensus.cc:3037] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.796936 5843 raft_consensus.cc:374] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.797119 5843 raft_consensus.cc:729] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.797458 5843 consensus_queue.cc:260] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.800675 5843 ts_tablet_manager.cc:1398] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:12:04.801043 5843 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.803385 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.804312 5654 leader_election.cc:341] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.805634 5843 tablet_bootstrap.cc:654] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.807165 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c83fcbd0cded4b18aa24693989f2f9ef" } new_table_name: "default.Table_f218240b99734c72a1ef75b6e7bb6534" modify_external_catalogs: true
I20240507 08:12:04.808509 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ab3f5d9e0ac34429a34b778d025e25f1" } new_table_name: "default.Table_45860b6e7d9d4788ba6c496b17838783" modify_external_catalogs: true
I20240507 08:12:04.809648 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_28e3986bcadb49cb861bd00e92c44f6b" } new_table_name: "default.Table_1b2f9330f0894e1c9a8b8640ab8b063a" modify_external_catalogs: true
W20240507 08:12:04.810267 5378 consensus_peers.cc:485] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:04.810832 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d692c7e6177c4bf487d2f91bbe15a9d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.811594 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cc196bc1f789402ab1d5fbb907534f09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.813385 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.815445 5580 raft_consensus.cc:3037] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.821985 6614 raft_consensus.cc:504] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.822364 6633 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:04.823835 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:04.824404 5378 leader_election.cc:341] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.825152 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.825654 5377 leader_election.cc:341] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.825798 5377 leader_election.cc:302] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.826215 6614 leader_election.cc:288] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.826507 6614 raft_consensus.cc:2726] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.826454 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c373339c2d25449e8c18ce24b35e434e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.828191 5840 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.828411 5840 ts_tablet_manager.cc:1367] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.004s
W20240507 08:12:04.829545 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82a95b4a3799468e85051bae28c2cdd3 consensus state has no leader
I20240507 08:12:04.829749 5840 raft_consensus.cc:348] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.830031 5840 raft_consensus.cc:374] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.830160 5840 raft_consensus.cc:729] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.830473 5840 consensus_queue.cc:260] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.831581 5840 ts_tablet_manager.cc:1398] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:04.832172 5840 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.834049 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:04.835600 6513 ts_tablet_manager.cc:1888] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:04.835798 6513 log.cc:1196] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/973a199a533b4bbea3c2615acdbf95ca
I20240507 08:12:04.839637 5839 tablet_bootstrap.cc:492] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:04.840240 6603 raft_consensus.cc:482] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.840487 6603 raft_consensus.cc:504] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:04.841117 6603 leader_election.cc:288] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.842803 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet fde460fdcf254af6a3cdaaac22ac2d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.842876 5839 ts_tablet_manager.cc:1367] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.003s
W20240507 08:12:04.843078 5520 leader_election.cc:341] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:12:04.843286 6513 ts_tablet_manager.cc:1909] T 973a199a533b4bbea3c2615acdbf95ca P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:04.843834 5840 tablet_bootstrap.cc:654] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.843722 5839 raft_consensus.cc:348] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.843914 5839 raft_consensus.cc:374] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.843966 5839 raft_consensus.cc:729] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.844179 5839 consensus_queue.cc:260] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.845067 5839 ts_tablet_manager.cc:1398] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 08:12:04.845433 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1944cb716e7b498a8f13a78b14d8c933 consensus state has no leader
I20240507 08:12:04.846735 5839 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:04.849704 6553 raft_consensus.cc:482] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.849828 6553 raft_consensus.cc:504] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.850411 6553 leader_election.cc:288] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.851045 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.851089 6684 raft_consensus.cc:482] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.851239 5443 raft_consensus.cc:2454] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:04.851244 6684 raft_consensus.cc:504] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.851663 5656 leader_election.cc:302] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:04.851758 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]). Expected version 1 got 0
I20240507 08:12:04.851905 6684 leader_election.cc:288] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.852160 6553 raft_consensus.cc:2781] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.852233 6553 raft_consensus.cc:482] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.852286 6553 raft_consensus.cc:3037] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.853546 6684 consensus_queue.cc:1035] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:04.853639 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.856034 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c373339c2d25449e8c18ce24b35e434e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.856076 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc196bc1f789402ab1d5fbb907534f09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.856029 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d692c7e6177c4bf487d2f91bbe15a9d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.859737 5377 leader_election.cc:341] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.861394 6675 raft_consensus.cc:504] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.864933 6675 leader_election.cc:288] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.865746 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:04.865931 5443 raft_consensus.cc:3037] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.869413 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e72c7ad89d294c4caba3c321f723cf2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.870119 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72c7ad89d294c4caba3c321f723cf2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.870410 6684 raft_consensus.cc:482] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.870540 6684 raft_consensus.cc:504] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.871199 6684 leader_election.cc:288] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:04.872363 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.874362 5377 leader_election.cc:341] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.874791 5840 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.874917 5840 ts_tablet_manager.cc:1367] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.006s
I20240507 08:12:04.875622 5840 raft_consensus.cc:348] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.875811 5840 raft_consensus.cc:374] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.875859 5840 raft_consensus.cc:729] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.876032 5840 consensus_queue.cc:260] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.876472 5840 ts_tablet_manager.cc:1398] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240507 08:12:04.876766 5840 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.878330 5840 tablet_bootstrap.cc:654] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.879482 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_afca694c460546b890018320967f9928"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.880025 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_afca694c460546b890018320967f9928 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.891813 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_452e7072fe42490f92423745f0bf9d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.893072 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_14b7abf57dc043ceb834e7e62ff05288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:04.893668 6614 raft_consensus.cc:482] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.893852 6614 raft_consensus.cc:504] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.894022 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d0b3de590398468abb0a9e5277b608c9" } new_table_name: "default.Table_0bfaacfdf3354499882aa8bdcc370cac" modify_external_catalogs: true
I20240507 08:12:04.895466 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:04.895987 5377 leader_election.cc:341] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.897233 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_452e7072fe42490f92423745f0bf9d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.898630 6614 leader_election.cc:288] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:04.899430 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14b7abf57dc043ceb834e7e62ff05288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.900640 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet accf88112b0b408e8d24828fc93055ef consensus state has no leader
I20240507 08:12:04.901996 6675 raft_consensus.cc:482] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.902189 6675 raft_consensus.cc:504] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.903129 6675 leader_election.cc:288] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.903507 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:04.903723 5444 raft_consensus.cc:2454] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:04.904157 5656 leader_election.cc:302] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:04.904209 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:04.904388 6675 raft_consensus.cc:2781] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:04.904508 6675 raft_consensus.cc:482] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:04.904649 6675 raft_consensus.cc:3037] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.850934 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.906536 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c48da19c9b9e4207b267137d240adc5b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:04.907121 6663 tablet.cc:1722] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:04.910197 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.910398 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f2b43daab6b4deeac94fc6f1ff34a76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:04.912171 5654 leader_election.cc:341] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:04.912550 5654 leader_election.cc:341] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.912516 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.913218 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:04.919095 5517 leader_election.cc:341] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:12:04.919294 5839 tablet_bootstrap.cc:654] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:04.919157 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.920972 5843 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:04.921097 5843 ts_tablet_manager.cc:1367] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.120s user 0.014s sys 0.014s
I20240507 08:12:04.921844 5517 leader_election.cc:302] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:04.921819 5843 raft_consensus.cc:348] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.922003 5843 raft_consensus.cc:374] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.922051 5843 raft_consensus.cc:729] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.922451 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9328db0568cb47a2a28f09d770d78ab6" } modify_external_catalogs: true
I20240507 08:12:04.922638 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.922663 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9328db0568cb47a2a28f09d770d78ab6" } modify_external_catalogs: true
I20240507 08:12:04.922749 6603 raft_consensus.cc:2726] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.922281 5843 consensus_queue.cc:260] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:04.922995 5378 leader_election.cc:341] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.923097 5378 leader_election.cc:302] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.923156 5843 ts_tablet_manager.cc:1398] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:04.923187 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:04.923583 5843 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:04.923560 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 88a2cfb5f046434c8340234a447a7e65 with delete_type TABLET_DATA_DELETED (replaced by tablet d451d03349794726a71c255a7f177436) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:04.924098 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1682cef551e4416a88a682fe5c7f4081" } new_table_name: "default.Table_687fbbf6bfdc4aab91a289e90561fafd" modify_external_catalogs: true
W20240507 08:12:04.924464 5378 leader_election.cc:341] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.924557 5378 leader_election.cc:302] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.924888 5843 tablet_bootstrap.cc:654] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:04.924913 5378 leader_election.cc:341] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.924985 5378 leader_election.cc:302] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.925328 6497 raft_consensus.cc:2726] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.925478 6497 raft_consensus.cc:2726] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.925499 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_10e42a0526054267b35b1acd7a1bc12c" } modify_external_catalogs: true
I20240507 08:12:04.925637 6497 raft_consensus.cc:2726] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:04.925665 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_10e42a0526054267b35b1acd7a1bc12c" } modify_external_catalogs: true
W20240507 08:12:04.926347 5654 leader_election.cc:341] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:04.927898 6547 tablet_replica.cc:330] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:04.928182 6547 raft_consensus.cc:2229] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.930215 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet bde62374dc7f4a67a071b6b658e501c4 (table default.Table_031f5e02ed6143dfa8d9d5dd9eff0cc3 [id=d0a57f2571ca478084824a44922ac99b]) successfully deleted
W20240507 08:12:04.930469 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ccc3e67eeb34c42b36edde7bacd23a5 already in progress: creating tablet
W20240507 08:12:04.930703 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d8a297a72b474e508184e1f32a2a37cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8a297a72b474e508184e1f32a2a37cf already in progress: creating tablet
W20240507 08:12:04.930913 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bc4c56d06c6c4c9f83495859be8a5c5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4c56d06c6c4c9f83495859be8a5c5e already in progress: creating tablet
W20240507 08:12:04.931078 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5643e8bc0add4df885536ff947d357c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5643e8bc0add4df885536ff947d357c6 already in progress: creating tablet
W20240507 08:12:04.931257 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7f18168519594979b02d651f093b2115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f18168519594979b02d651f093b2115 already in progress: creating tablet
W20240507 08:12:04.931430 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b27f89f6db6746fb870dad2a3fb27457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27f89f6db6746fb870dad2a3fb27457 already in progress: creating tablet
I20240507 08:12:04.941725 6513 tablet_replica.cc:330] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:04.941942 6513 raft_consensus.cc:2229] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:04.942056 6513 raft_consensus.cc:2258] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:04.942459 6513 ts_tablet_manager.cc:1875] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:04.945875 6553 raft_consensus.cc:504] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.946496 6553 leader_election.cc:288] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.946842 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 973a199a533b4bbea3c2615acdbf95ca (table default.Table_964368f3c7824528b36f3fb37bb2e215 [id=4ecddc64b3da4de5a70e7a211f9ea037]) successfully deleted
W20240507 08:12:04.947126 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ccc3e67eeb34c42b36edde7bacd23a5 already in progress: creating tablet
W20240507 08:12:04.947302 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bc4c56d06c6c4c9f83495859be8a5c5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc4c56d06c6c4c9f83495859be8a5c5e already in progress: creating tablet
W20240507 08:12:04.947467 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d8a297a72b474e508184e1f32a2a37cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8a297a72b474e508184e1f32a2a37cf already in progress: creating tablet
W20240507 08:12:04.947633 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5643e8bc0add4df885536ff947d357c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5643e8bc0add4df885536ff947d357c6 already in progress: creating tablet
W20240507 08:12:04.947851 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7f18168519594979b02d651f093b2115 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7f18168519594979b02d651f093b2115 already in progress: creating tablet
W20240507 08:12:04.948071 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b27f89f6db6746fb870dad2a3fb27457 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b27f89f6db6746fb870dad2a3fb27457 already in progress: creating tablet
I20240507 08:12:04.948647 5443 raft_consensus.cc:2454] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:04.950309 5656 leader_election.cc:302] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:04.950794 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:04.950973 5443 raft_consensus.cc:3037] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:04.951735 6603 raft_consensus.cc:482] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.951879 6603 raft_consensus.cc:504] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.953953 6553 raft_consensus.cc:2781] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:04.954110 6553 raft_consensus.cc:686] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:04.955493 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3dc49fd6841c420288aa1df3950a088b consensus state has no leader
I20240507 08:12:04.960044 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9" } modify_external_catalogs: true
I20240507 08:12:04.960343 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_861cfb7e51fb48468db2c8b4b0a1bbd9" } modify_external_catalogs: true
I20240507 08:12:04.960482 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:04.960780 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
W20240507 08:12:04.961246 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89db2847c09b4c74b809bd13f78d0b8b consensus state has no leader
I20240507 08:12:04.964733 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:04.965268 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_139fa8ac0afe4140ac71e91821a98481" } modify_external_catalogs: true
I20240507 08:12:04.965446 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_139fa8ac0afe4140ac71e91821a98481" } modify_external_catalogs: true
I20240507 08:12:04.967015 6553 consensus_queue.cc:237] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.971838 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a07279891494410096ed3d7da872b203" } modify_external_catalogs: true
I20240507 08:12:04.971961 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:04.972162 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a07279891494410096ed3d7da872b203" } modify_external_catalogs: true
I20240507 08:12:04.972158 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c91f44a911c74b99ae52369cf8d517a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.972810 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c91f44a911c74b99ae52369cf8d517a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.974961 6508 raft_consensus.cc:482] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:04.975100 6508 raft_consensus.cc:504] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.975651 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5
I20240507 08:12:04.975703 6508 leader_election.cc:288] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.976758 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:04.976831 5656 leader_election.cc:341] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: a7df45c39d1b490db97c72a0eb6cf3ba
I20240507 08:12:04.977453 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.982367 5255 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_d563809a6ab04242827f32b2b0a6a40f [id=c59b446f65c842478dae29b4065fce5a] alter complete (version 1)
I20240507 08:12:04.982509 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 96ef77ef4f474107bf31395f5f30c12f
I20240507 08:12:04.983220 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8a397c40278141b7bb80ea5fcdbb125b
I20240507 08:12:04.986029 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ed0018356e254b82995061ef7290556d
I20240507 08:12:04.986974 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16e0eb926aa043c89e47aa0786c20cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:04.987565 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16e0eb926aa043c89e47aa0786c20cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:04.988337 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef77ef4f474107bf31395f5f30c12f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.989243 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0018356e254b82995061ef7290556d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.989243 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef77ef4f474107bf31395f5f30c12f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.989296 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.990989 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.991024 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0018356e254b82995061ef7290556d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.992590 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4e3ea4658bb946439e4a50962444b3e2
I20240507 08:12:04.993371 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:04.993739 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 5782cfa5eca74c19a2e724a9124597f2 (table default.Table_ee5279fc19a046aebae1f622389f59a2 [id=c6a13f9902b0424ca8df2184779a407c]). Expected version 3 got 0
I20240507 08:12:04.994009 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]). Expected version 1 got 0
I20240507 08:12:04.994220 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:04.996217 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 55bacb9fd24b46709704eb852e051eb1 (DEFAULT_TABLE table=default.Table_8a36aab1e10b41c589b8b0eac87af0db [id=e0365a8f215c4f058f47daf0b4957f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.996819 5840 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:04.996951 5840 ts_tablet_manager.cc:1367] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.120s user 0.005s sys 0.008s
I20240507 08:12:04.997031 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 55bacb9fd24b46709704eb852e051eb1 (DEFAULT_TABLE table=default.Table_8a36aab1e10b41c589b8b0eac87af0db [id=e0365a8f215c4f058f47daf0b4957f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:04.997642 5840 raft_consensus.cc:348] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:04.997898 5840 raft_consensus.cc:374] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:04.997970 5840 raft_consensus.cc:729] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:04.998208 6603 leader_election.cc:288] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:04.998193 5840 consensus_queue.cc:260] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:04.998513 5517 leader_election.cc:341] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 072702ef2a6d43248132da6aa3a2d493
W20240507 08:12:04.998672 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 44498us
I20240507 08:12:04.999305 5840 ts_tablet_manager.cc:1398] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:04.999557 5840 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:04.999773 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e3ea4658bb946439e4a50962444b3e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:05.000260 5520 leader_election.cc:341] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 072702ef2a6d43248132da6aa3a2d493
I20240507 08:12:05.000396 5520 leader_election.cc:302] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.000857 6603 raft_consensus.cc:2726] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.001447 6547 raft_consensus.cc:2258] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.002068 6547 ts_tablet_manager.cc:1875] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.002770 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7df45c39d1b490db97c72a0eb6cf3ba" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.004281 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 55bacb9fd24b46709704eb852e051eb1 (DEFAULT_TABLE table=default.Table_8a36aab1e10b41c589b8b0eac87af0db [id=e0365a8f215c4f058f47daf0b4957f52]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.004698 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.004912 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 96ef77ef4f474107bf31395f5f30c12f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.004925 5840 tablet_bootstrap.cc:654] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.005093 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a397c40278141b7bb80ea5fcdbb125b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.005281 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet ed0018356e254b82995061ef7290556d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.005837 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:05.006366 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.006444 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.009719 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.009963 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:05.010529 5654 leader_election.cc:341] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.010655 5654 leader_election.cc:302] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.011335 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.011845 5654 leader_election.cc:341] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.013276 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.013670 6553 raft_consensus.cc:2726] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.013999 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bacb9fd24b46709704eb852e051eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.015112 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 62b8fb67327b4f779f3bb494c03616a3 (DEFAULT_TABLE table=default.Table_c083a659194342cdbdbb42898c701c6d [id=2dd0295898004cb2a11af3782be8102b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.015749 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 62b8fb67327b4f779f3bb494c03616a3 (DEFAULT_TABLE table=default.Table_c083a659194342cdbdbb42898c701c6d [id=2dd0295898004cb2a11af3782be8102b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.016371 5691 tablet_service.cc:1467] Processing CreateTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec (DEFAULT_TABLE table=default.Table_a77bc176ba0948c796f99b08bcc16d1d [id=85c7dd47571441e38bae1663fb034c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.017935 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 62b8fb67327b4f779f3bb494c03616a3 (DEFAULT_TABLE table=default.Table_c083a659194342cdbdbb42898c701c6d [id=2dd0295898004cb2a11af3782be8102b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.018287 5686 tablet_service.cc:1467] Processing CreateTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.018431 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b8fb67327b4f779f3bb494c03616a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.018781 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec (DEFAULT_TABLE table=default.Table_a77bc176ba0948c796f99b08bcc16d1d [id=85c7dd47571441e38bae1663fb034c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.019979 5843 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.020133 5843 ts_tablet_manager.cc:1367] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.003s
I20240507 08:12:05.021226 5843 raft_consensus.cc:348] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.021479 5843 raft_consensus.cc:374] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.021539 5843 raft_consensus.cc:729] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.021862 5843 consensus_queue.cc:260] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.022492 5549 tablet_service.cc:1467] Processing CreateTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec (DEFAULT_TABLE table=default.Table_a77bc176ba0948c796f99b08bcc16d1d [id=85c7dd47571441e38bae1663fb034c87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.022868 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c1bc3385134d8fbcc94458ef0612ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.021929 5422 tablet_service.cc:1467] Processing CreateTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.023880 5548 tablet_service.cc:1467] Processing CreateTablet for tablet bfa2139385d34c37a7ac14d78480a779 (DEFAULT_TABLE table=default.Table_c7be75b22b2d41878a28edd9a44b6456 [id=67c1eced0ba641e38c8b300c1e910209]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.024746 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa2139385d34c37a7ac14d78480a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.028743 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:05.023850 5565 tablet_service.cc:1467] Processing CreateTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.030000 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e2deaadfe34c2dbdc9819729cb4890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.030144 5843 ts_tablet_manager.cc:1398] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:12:05.031201 5698 tablet_service.cc:1467] Processing CreateTablet for tablet bfa2139385d34c37a7ac14d78480a779 (DEFAULT_TABLE table=default.Table_c7be75b22b2d41878a28edd9a44b6456 [id=67c1eced0ba641e38c8b300c1e910209]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.031826 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3 (DEFAULT_TABLE table=default.Table_42ec5a6050fd445ebe2a0545cc26cbc0 [id=dd7e211551e24b89bc41d6b08d94eef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.032202 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.033947 5427 tablet_service.cc:1467] Processing CreateTablet for tablet bfa2139385d34c37a7ac14d78480a779 (DEFAULT_TABLE table=default.Table_c7be75b22b2d41878a28edd9a44b6456 [id=67c1eced0ba641e38c8b300c1e910209]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.034497 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c62499ccee34036b23e4fe0804fd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.038640 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3 (DEFAULT_TABLE table=default.Table_42ec5a6050fd445ebe2a0545cc26cbc0 [id=dd7e211551e24b89bc41d6b08d94eef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.038640 5417 tablet_service.cc:1467] Processing CreateTablet for tablet f416d08748ac483ab0ce98240e9eb036 (DEFAULT_TABLE table=default.Table_5b668756389c48ef943681187557167a [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.040124 5841 ts_tablet_manager.cc:1888] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.040222 5841 log.cc:1196] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e555a89d340448bc852f2441ca850213
I20240507 08:12:05.040519 5841 ts_tablet_manager.cc:1909] T e555a89d340448bc852f2441ca850213 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:05.040760 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bacb9fd24b46709704eb852e051eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.041096 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:05.041316 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c62499ccee34036b23e4fe0804fd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.041776 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b8fb67327b4f779f3bb494c03616a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.041869 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55bacb9fd24b46709704eb852e051eb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.041865 5550 tablet_service.cc:1467] Processing CreateTablet for tablet f416d08748ac483ab0ce98240e9eb036 (DEFAULT_TABLE table=default.Table_5b668756389c48ef943681187557167a [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.042167 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c1bc3385134d8fbcc94458ef0612ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.042199 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f416d08748ac483ab0ce98240e9eb036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.042343 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62b8fb67327b4f779f3bb494c03616a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.042543 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e2deaadfe34c2dbdc9819729cb4890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.042742 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5c1bc3385134d8fbcc94458ef0612ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.042906 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa2139385d34c37a7ac14d78480a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.043347 5443 raft_consensus.cc:2454] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:05.044338 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3 (DEFAULT_TABLE table=default.Table_42ec5a6050fd445ebe2a0545cc26cbc0 [id=dd7e211551e24b89bc41d6b08d94eef2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.044394 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0b39549076654463a9689c30a3cb49b1" } new_table_name: "default.Table_95142d176f0548108c4ccfe30d8da018" modify_external_catalogs: true
I20240507 08:12:05.044662 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.045707 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c62499ccee34036b23e4fe0804fd2ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.045749 6675 raft_consensus.cc:504] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.045840 5693 tablet_service.cc:1467] Processing CreateTablet for tablet f416d08748ac483ab0ce98240e9eb036 (DEFAULT_TABLE table=default.Table_5b668756389c48ef943681187557167a [id=5331470f800a425eb58c185aef9bd183]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.046115 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f416d08748ac483ab0ce98240e9eb036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.046499 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f416d08748ac483ab0ce98240e9eb036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.046597 6675 leader_election.cc:288] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.046718 6675 raft_consensus.cc:482] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.046805 6675 raft_consensus.cc:504] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.047252 6675 leader_election.cc:288] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.047813 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bfa2139385d34c37a7ac14d78480a779. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.047960 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8079bcaf8a7e43d795aa8b3e72a91cc5" } new_table_name: "default.Table_4a3987634ddc4464a1525c81bdb9d050" modify_external_catalogs: true
I20240507 08:12:05.048207 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e2deaadfe34c2dbdc9819729cb4890. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.049264 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2ecc11b84a324aaa9c19ac2f270ea9c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.050073 5656 leader_election.cc:302] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.050479 6553 raft_consensus.cc:2781] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.050592 6553 raft_consensus.cc:686] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:05.050824 6553 consensus_queue.cc:237] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.051990 5547 tablet_service.cc:1467] Processing CreateTablet for tablet c40ed1afee294143ab85a8c03d080b74 (DEFAULT_TABLE table=default.Table_b81673c2cce64edba3bbf1a45e6c3e49 [id=bdd1c4ed03f9454ebb97f3fca4b4adb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.052676 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.052877 5444 raft_consensus.cc:2454] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.053368 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.053520 5443 raft_consensus.cc:3037] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.053843 5656 leader_election.cc:302] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:05.055852 6675 raft_consensus.cc:2781] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.055939 6675 raft_consensus.cc:482] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.055994 6675 raft_consensus.cc:3037] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.056821 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.056818 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.059254 5654 leader_election.cc:341] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:05.059576 5654 leader_election.cc:341] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.068614 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_816c7491958e48f29ae98e8b458aa440"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.069859 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_816c7491958e48f29ae98e8b458aa440 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.070628 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dec376d239c14388b1cdd9fa9b67873f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.071157 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dec376d239c14388b1cdd9fa9b67873f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.072328 5839 tablet_bootstrap.cc:492] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.072491 5839 ts_tablet_manager.cc:1367] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.226s user 0.007s sys 0.016s
I20240507 08:12:05.072964 6663 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:05.073181 5839 raft_consensus.cc:348] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.073393 5839 raft_consensus.cc:374] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.073483 5839 raft_consensus.cc:729] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:05.074913 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648113d9b8164cc59ba238ace8f65b39 consensus state has no leader
I20240507 08:12:05.076179 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_73e379813c044602aa406c9ac47557ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.076678 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73e379813c044602aa406c9ac47557ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.078991 6614 raft_consensus.cc:482] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.079141 6614 raft_consensus.cc:504] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.079977 6614 leader_election.cc:288] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.080587 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:05.081173 5377 leader_election.cc:341] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.081763 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.082459 5378 leader_election.cc:341] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.082551 5378 leader_election.cc:302] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.083360 6614 raft_consensus.cc:2726] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.087141 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_35569f4031ae4c15a8f2653100e5551b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.087738 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35569f4031ae4c15a8f2653100e5551b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.089215 5262 catalog_manager.cc:5484] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:05.089718 5262 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 5782cfa5eca74c19a2e724a9124597f2 (table default.Table_ee5279fc19a046aebae1f622389f59a2 [id=c6a13f9902b0424ca8df2184779a407c]). Expected version 3 got 0
I20240507 08:12:05.089874 5255 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_29880271f0474162818f1334e42c2a3d [id=95b44cbcf26f489c8f4d8e07cc615776]). Expected version 1 got 0
I20240507 08:12:05.093796 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f5289d2de1e64222b8a1faf354cf0048"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.094380 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5289d2de1e64222b8a1faf354cf0048 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:05.098899 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2717b2d928674c17809ef4b0141e352e consensus state has no leader
I20240507 08:12:05.102192 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_299103db24834157bce006563e9b48a3" } new_table_name: "default.Table_02c584c9b6774d37b485b35ba37050ec" modify_external_catalogs: true
I20240507 08:12:05.108569 5843 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.110083 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0bece17329174cf886f812c040ec88df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.110734 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bece17329174cf886f812c040ec88df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.110792 5843 tablet_bootstrap.cc:654] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.112802 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_493ded421178438798c30ff021b4531b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.113504 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_493ded421178438798c30ff021b4531b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.117679 6553 raft_consensus.cc:482] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.117849 6553 raft_consensus.cc:504] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.118490 6553 leader_election.cc:288] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.119431 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.119647 5444 raft_consensus.cc:2454] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.120086 5656 leader_election.cc:302] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:05.120374 6553 raft_consensus.cc:2781] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.120447 6553 raft_consensus.cc:482] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.120514 6553 raft_consensus.cc:3037] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.127854 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.130285 5654 leader_election.cc:341] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.130316 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 47a154769d0f4501837888feba98f33d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.132452 5839 consensus_queue.cc:260] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.134510 6675 raft_consensus.cc:504] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.135056 6513 ts_tablet_manager.cc:1888] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.135128 6675 leader_election.cc:288] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.135171 6513 log.cc:1196] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:05.135455 6513 ts_tablet_manager.cc:1909] T 1cd02a49359743c4b4e540ab0338d8ed P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:05.137787 5443 raft_consensus.cc:2454] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:05.138226 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40ed1afee294143ab85a8c03d080b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.138962 5839 ts_tablet_manager.cc:1398] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.066s user 0.003s sys 0.000s
I20240507 08:12:05.139506 5841 tablet_replica.cc:330] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:05.139693 5841 raft_consensus.cc:2229] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.139940 5841 raft_consensus.cc:2258] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.140468 5841 ts_tablet_manager.cc:1875] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.142469 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e555a89d340448bc852f2441ca850213 (table default.Table_c207fdb532c4420b8289d1d9959722dd [id=a19f35cb006a4367961f26227bc0a662]) successfully deleted
I20240507 08:12:05.143954 5656 leader_election.cc:302] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:05.144716 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe7a7c1bee444c75a4ed6215db0e25bd has no consensus state
I20240507 08:12:05.144920 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.145115 5443 raft_consensus.cc:3037] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.146487 6675 raft_consensus.cc:2781] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.146624 6675 raft_consensus.cc:686] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:05.146872 6675 consensus_queue.cc:237] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.148692 5839 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.149950 5414 tablet_service.cc:1467] Processing CreateTablet for tablet c40ed1afee294143ab85a8c03d080b74 (DEFAULT_TABLE table=default.Table_b81673c2cce64edba3bbf1a45e6c3e49 [id=bdd1c4ed03f9454ebb97f3fca4b4adb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.150477 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40ed1afee294143ab85a8c03d080b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.150625 5839 tablet_bootstrap.cc:654] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.159191 5411 tablet_service.cc:1467] Processing CreateTablet for tablet bd43e0da12154110a5128e418ace04f4 (DEFAULT_TABLE table=default.Table_91b52ae3fbb140a7bad7e02bd6e58c18 [id=33f4060b84294483aea94adaa9fa1098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.159592 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43e0da12154110a5128e418ace04f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.159538 5416 tablet_service.cc:1467] Processing CreateTablet for tablet aafd1dc2eaf84390b072f01a9e76232d (DEFAULT_TABLE table=default.Table_b8da67c846954c4a95c5a20759624d6f [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.160137 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 4776584f1c6e43368217b604a64e384f (DEFAULT_TABLE table=default.Table_6709887aaee84c1ba633b7f910c797e2 [id=aa10cd958b784d8ba6ddb2dd4fa9c012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.160486 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafd1dc2eaf84390b072f01a9e76232d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.162117 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 137a9571d7564916aed755590747e0e3 (DEFAULT_TABLE table=default.Table_7f5a117d33fe42ae985e25ef355c1def [id=ec9367d300e74143923f0f7705f4c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.162427 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137a9571d7564916aed755590747e0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.168042 5411 tablet_service.cc:1467] Processing CreateTablet for tablet f89efb8414dc49369d361adf908927e6 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.168468 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89efb8414dc49369d361adf908927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.168713 6513 tablet_replica.cc:330] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:05.168957 6513 raft_consensus.cc:2229] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.169169 6513 raft_consensus.cc:2258] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.169775 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4776584f1c6e43368217b604a64e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.170966 5840 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.171115 5840 ts_tablet_manager.cc:1367] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.172s user 0.026s sys 0.012s
I20240507 08:12:05.171816 5840 raft_consensus.cc:348] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.172005 5840 raft_consensus.cc:374] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.172055 5840 raft_consensus.cc:729] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.172240 5840 consensus_queue.cc:260] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.177207 5693 tablet_service.cc:1467] Processing CreateTablet for tablet c40ed1afee294143ab85a8c03d080b74 (DEFAULT_TABLE table=default.Table_b81673c2cce64edba3bbf1a45e6c3e49 [id=bdd1c4ed03f9454ebb97f3fca4b4adb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.177606 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c40ed1afee294143ab85a8c03d080b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.178267 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 4776584f1c6e43368217b604a64e384f (DEFAULT_TABLE table=default.Table_6709887aaee84c1ba633b7f910c797e2 [id=aa10cd958b784d8ba6ddb2dd4fa9c012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.180682 5840 ts_tablet_manager.cc:1398] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:05.181475 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 4776584f1c6e43368217b604a64e384f (DEFAULT_TABLE table=default.Table_6709887aaee84c1ba633b7f910c797e2 [id=aa10cd958b784d8ba6ddb2dd4fa9c012]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.181804 5840 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.181825 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4776584f1c6e43368217b604a64e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.182308 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 137a9571d7564916aed755590747e0e3 (DEFAULT_TABLE table=default.Table_7f5a117d33fe42ae985e25ef355c1def [id=ec9367d300e74143923f0f7705f4c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.182612 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137a9571d7564916aed755590747e0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.185643 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 137a9571d7564916aed755590747e0e3 (DEFAULT_TABLE table=default.Table_7f5a117d33fe42ae985e25ef355c1def [id=ec9367d300e74143923f0f7705f4c2b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.185992 5690 tablet_service.cc:1467] Processing CreateTablet for tablet bd43e0da12154110a5128e418ace04f4 (DEFAULT_TABLE table=default.Table_91b52ae3fbb140a7bad7e02bd6e58c18 [id=33f4060b84294483aea94adaa9fa1098]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.186321 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43e0da12154110a5128e418ace04f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.186378 5683 tablet_service.cc:1467] Processing CreateTablet for tablet aafd1dc2eaf84390b072f01a9e76232d (DEFAULT_TABLE table=default.Table_b8da67c846954c4a95c5a20759624d6f [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.186407 5565 tablet_service.cc:1467] Processing CreateTablet for tablet aafd1dc2eaf84390b072f01a9e76232d (DEFAULT_TABLE table=default.Table_b8da67c846954c4a95c5a20759624d6f [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.186708 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafd1dc2eaf84390b072f01a9e76232d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.187081 5549 tablet_service.cc:1467] Processing CreateTablet for tablet f89efb8414dc49369d361adf908927e6 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.190986 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1b2f9330f0894e1c9a8b8640ab8b063a" } new_table_name: "default.Table_46f6ad9b630e4b26995175a360c78187" modify_external_catalogs: true
W20240507 08:12:05.191723 5654 consensus_peers.cc:485] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.191767 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7c7240e1d89c4c05a124c7126954e1ae" } modify_external_catalogs: true
I20240507 08:12:05.191944 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7c7240e1d89c4c05a124c7126954e1ae" } modify_external_catalogs: true
I20240507 08:12:05.193284 5843 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.193406 5843 ts_tablet_manager.cc:1367] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20240507 08:12:05.193882 5840 tablet_bootstrap.cc:654] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.194051 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1cd02a49359743c4b4e540ab0338d8ed (table default.Table_75283770e2c44851bcfa380f6385359d [id=fab372c29d234af7b0bfdae99be63919]) successfully deleted
I20240507 08:12:05.194530 5843 raft_consensus.cc:348] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.194707 5843 raft_consensus.cc:374] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.194752 5843 raft_consensus.cc:729] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.194975 5843 consensus_queue.cc:260] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.196694 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 055a30bfc61b4ff2be9f2418ca59b030 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.198117 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_20e0ec532f1549dd9c1df7c636346293" } modify_external_catalogs: true
I20240507 08:12:05.198284 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_20e0ec532f1549dd9c1df7c636346293" } modify_external_catalogs: true
I20240507 08:12:05.199103 6513 ts_tablet_manager.cc:1875] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.200356 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa" } modify_external_catalogs: true
I20240507 08:12:05.200543 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9dd1f3b5ae6e4ccf8455601ce62a05aa" } modify_external_catalogs: true
I20240507 08:12:05.200951 5843 ts_tablet_manager.cc:1398] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:05.203953 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.204403 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_be93d8cabe2f4a27b5a47dddbb37b42e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.205688 5555 tablet_service.cc:1467] Processing CreateTablet for tablet bd43e0da12154110a5128e418ace04f4 (DEFAULT_TABLE table=default.Table_91b52ae3fbb140a7bad7e02bd6e58c18 [id=33f4060b84294483aea94adaa9fa1098]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.206303 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be93d8cabe2f4a27b5a47dddbb37b42e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.206779 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b19f64f84d434d8684f3b784794f2f52" } modify_external_catalogs: true
I20240507 08:12:05.207314 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b19f64f84d434d8684f3b784794f2f52" } modify_external_catalogs: true
I20240507 08:12:05.208858 5840 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.208989 5840 ts_tablet_manager.cc:1367] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20240507 08:12:05.209462 5843 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.209801 5840 raft_consensus.cc:348] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.209986 5840 raft_consensus.cc:374] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.210044 5840 raft_consensus.cc:729] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.211050 5843 tablet_bootstrap.cc:654] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.210990 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08aac6e88b7c41d8a59d1e12e7f6c14e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.211606 5654 leader_election.cc:341] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.213734 6603 raft_consensus.cc:482] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.213903 6603 raft_consensus.cc:504] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.214659 6603 leader_election.cc:288] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.215334 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3dc5823424924e0bbff545ee187770d8" } modify_external_catalogs: true
I20240507 08:12:05.215410 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.215556 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3dc5823424924e0bbff545ee187770d8" } modify_external_catalogs: true
I20240507 08:12:05.216310 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3e8b4d5d594f378451009c313f6bfa" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.216846 5517 leader_election.cc:302] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.217298 6603 raft_consensus.cc:2726] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.217350 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8dc4bd163dcc4d21bdade90ef02ceeaf" } new_table_name: "default.Table_d228b6629c7349a5843e85ab6a55e3bd" modify_external_catalogs: true
I20240507 08:12:05.220430 5259 catalog_manager.cc:5484] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:05.220921 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]). Expected version 1 got 0
I20240507 08:12:05.225739 5840 consensus_queue.cc:260] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.226614 5840 ts_tablet_manager.cc:1398] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20240507 08:12:05.227506 5840 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.227895 5843 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.228031 5843 ts_tablet_manager.cc:1367] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.008s
I20240507 08:12:05.229449 6508 consensus_queue.cc:1035] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:05.230310 5843 raft_consensus.cc:348] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.230372 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 5c34a8ee0ca24e9490f3ba409b62a009 (DEFAULT_TABLE table=default.Table_2b0061a7e41c4d66aaff6de6bad0e892 [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.230509 5843 raft_consensus.cc:374] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.230557 5843 raft_consensus.cc:729] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.230770 5843 consensus_queue.cc:260] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.231943 5682 tablet_service.cc:1467] Processing CreateTablet for tablet f89efb8414dc49369d361adf908927e6 (DEFAULT_TABLE table=default.Table_a99b17a89c46447ea9e6544b2f7eb4d4 [id=b591a06a8e02403ab74223596a6f8acb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.231914 6553 raft_consensus.cc:504] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.232211 5443 raft_consensus.cc:2454] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:05.232667 6553 leader_election.cc:288] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.233435 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c34a8ee0ca24e9490f3ba409b62a009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.236048 5840 tablet_bootstrap.cc:654] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.237478 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.237686 5443 raft_consensus.cc:3037] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.237632 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fcc1d7e309946d783a524b61d5f1269" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.238214 5654 leader_election.cc:341] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.240218 5656 leader_election.cc:302] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.245682 6553 raft_consensus.cc:2781] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.245826 6553 raft_consensus.cc:686] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:05.246110 6553 consensus_queue.cc:237] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.246378 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89efb8414dc49369d361adf908927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.250155 6676 raft_consensus.cc:482] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.250285 6676 raft_consensus.cc:504] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.250911 6676 leader_election.cc:288] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.254362 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 (DEFAULT_TABLE table=default.Table_5e958b4da1dd43ec9cc03a7554860b1c [id=d5f4df1868304662baa91b19c1726746]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.259603 5843 ts_tablet_manager.cc:1398] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240507 08:12:05.260087 5843 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.261801 5843 tablet_bootstrap.cc:654] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.268946 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:05.273810 5377 leader_election.cc:341] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.284005 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.286473 5378 leader_election.cc:341] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.286620 5378 leader_election.cc:302] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.286986 6676 raft_consensus.cc:2726] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.289438 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 5c34a8ee0ca24e9490f3ba409b62a009 (DEFAULT_TABLE table=default.Table_2b0061a7e41c4d66aaff6de6bad0e892 [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.290367 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 (DEFAULT_TABLE table=default.Table_5e958b4da1dd43ec9cc03a7554860b1c [id=d5f4df1868304662baa91b19c1726746]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.300961 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9846e0eab74388af231e9942a6881a (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.302634 5843 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.302795 5843 ts_tablet_manager.cc:1367] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.014s
I20240507 08:12:05.303694 5840 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.303620 5843 raft_consensus.cc:348] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.303822 5843 raft_consensus.cc:374] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.303843 5840 ts_tablet_manager.cc:1367] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.076s user 0.018s sys 0.015s
I20240507 08:12:05.303910 5843 raft_consensus.cc:729] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.304189 5843 consensus_queue.cc:260] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.304514 5840 raft_consensus.cc:348] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.304698 5840 raft_consensus.cc:374] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.304752 5840 raft_consensus.cc:729] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.304921 5840 consensus_queue.cc:260] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.305680 5839 tablet_bootstrap.cc:492] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.305889 5839 ts_tablet_manager.cc:1367] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.157s user 0.003s sys 0.000s
I20240507 08:12:05.307031 5840 ts_tablet_manager.cc:1398] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:05.307312 5840 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.309183 5839 raft_consensus.cc:348] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.309466 5839 raft_consensus.cc:374] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.309612 5839 raft_consensus.cc:729] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.309885 5839 consensus_queue.cc:260] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.310467 5840 tablet_bootstrap.cc:654] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.312687 5839 ts_tablet_manager.cc:1398] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:05.313505 5839 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.314673 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9846e0eab74388af231e9942a6881a (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.314903 6676 raft_consensus.cc:482] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.314913 5424 tablet_service.cc:1467] Processing CreateTablet for tablet e3783fc7fe434696898ba14ab77a9058 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.315090 6676 raft_consensus.cc:504] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.315624 5839 tablet_bootstrap.cc:654] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.315763 6676 leader_election.cc:288] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.316336 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.316504 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.317027 5377 leader_election.cc:341] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:05.317248 5378 leader_election.cc:341] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.317353 5378 leader_election.cc:302] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:05.317307 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 91691.5us
I20240507 08:12:05.317656 6684 raft_consensus.cc:482] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.317777 6684 raft_consensus.cc:504] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.318271 6684 leader_election.cc:288] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.318395 6684 raft_consensus.cc:2726] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:05.319334 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1944cb716e7b498a8f13a78b14d8c933 consensus state has no leader
I20240507 08:12:05.320336 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3769eb670ff442e2ad2fa5b6a934512a
I20240507 08:12:05.322693 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.322793 5843 ts_tablet_manager.cc:1398] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20240507 08:12:05.322963 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.323047 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 (DEFAULT_TABLE table=default.Table_5e958b4da1dd43ec9cc03a7554860b1c [id=d5f4df1868304662baa91b19c1726746]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.323122 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.323331 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 5c34a8ee0ca24e9490f3ba409b62a009 (DEFAULT_TABLE table=default.Table_2b0061a7e41c4d66aaff6de6bad0e892 [id=5714d419aa014285ac6779ef49cc29fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.323909 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9846e0eab74388af231e9942a6881a (DEFAULT_TABLE table=default.Table_996bd171383346f6b81bdcfc316f9ee4 [id=35b2da7f60ee4341b27b59b8ec9e773c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.324636 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.326046 5378 leader_election.cc:341] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.326543 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 3769eb670ff442e2ad2fa5b6a934512a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:05.326843 5377 leader_election.cc:341] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.327013 5377 leader_election.cc:302] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.329946 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 055a30bfc61b4ff2be9f2418ca59b030
I20240507 08:12:05.330513 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 055a30bfc61b4ff2be9f2418ca59b030 deleted and replaced by tablet 38ce7a166001486197b78f8b02a437ed
I20240507 08:12:05.330655 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4ad95383487e458ca162038f6940271b
I20240507 08:12:05.330870 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet b5d4993c287a40cf964877b5464372a6 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.331230 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16e0eb926aa043c89e47aa0786c20cd8" } new_table_name: "default.Table_78c7e392d6e644cea15ddcdcb2e991aa" modify_external_catalogs: true
I20240507 08:12:05.331507 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet d451d03349794726a71c255a7f177436 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.332466 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b6ba3397e33b4cc7805a25e7535a383a
I20240507 08:12:05.332975 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet 2b13fbddcab5460db69f43e63f490fba from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.333051 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2bb648fe98e74248b5cd65603de58a02
I20240507 08:12:05.333634 5841 ts_tablet_manager.cc:1888] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.333766 5841 log.cc:1196] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:05.334121 5841 ts_tablet_manager.cc:1909] T d45d3fbbe3764261a3a24523a77c6443 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:05.333956 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c3cca313bfc24efcacf2454a66c6e114"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.334367 5443 raft_consensus.cc:2454] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:05.334625 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3cca313bfc24efcacf2454a66c6e114 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.334846 6547 ts_tablet_manager.cc:1888] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.334975 6547 log.cc:1196] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f5e48a2b4a0e4088ad009a7f8fe77f3f
I20240507 08:12:05.335162 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3397e33b4cc7805a25e7535a383a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.335295 6547 ts_tablet_manager.cc:1909] T f5e48a2b4a0e4088ad009a7f8fe77f3f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:05.335377 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb648fe98e74248b5cd65603de58a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.336164 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.337378 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_34e4975648c24bb59e66757e3319a0a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.337854 6513 ts_tablet_manager.cc:1888] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.337953 6513 log.cc:1196] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/a7df45c39d1b490db97c72a0eb6cf3ba
W20240507 08:12:05.337968 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34e4975648c24bb59e66757e3319a0a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.338250 6513 ts_tablet_manager.cc:1909] T a7df45c39d1b490db97c72a0eb6cf3ba P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:05.338932 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad95383487e458ca162038f6940271b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.339197 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_75b3c3cdeef54143a244f17d717d4e17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.339717 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75b3c3cdeef54143a244f17d717d4e17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.341647 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb648fe98e74248b5cd65603de58a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.342864 5560 tablet_service.cc:1467] Processing CreateTablet for tablet e3783fc7fe434696898ba14ab77a9058 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.343385 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad95383487e458ca162038f6940271b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.343576 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3397e33b4cc7805a25e7535a383a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.343806 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.343924 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3783fc7fe434696898ba14ab77a9058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.344014 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.344223 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ad95383487e458ca162038f6940271b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.344398 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet b6ba3397e33b4cc7805a25e7535a383a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.344483 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fbe0f58bad541e7bcfb611105328d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.344574 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb648fe98e74248b5cd65603de58a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.344883 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c34a8ee0ca24e9490f3ba409b62a009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.345420 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9846e0eab74388af231e9942a6881a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.345685 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd43e0da12154110a5128e418ace04f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.346968 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c34a8ee0ca24e9490f3ba409b62a009. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.347440 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fbe0f58bad541e7bcfb611105328d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.347858 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9846e0eab74388af231e9942a6881a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.350194 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.353837 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4776584f1c6e43368217b604a64e384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.354355 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 137a9571d7564916aed755590747e0e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.354779 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aafd1dc2eaf84390b072f01a9e76232d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.355165 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f89efb8414dc49369d361adf908927e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.357280 6553 raft_consensus.cc:482] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.357406 6553 raft_consensus.cc:504] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.358069 6553 leader_election.cc:288] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.358950 5656 leader_election.cc:302] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.359100 5690 tablet_service.cc:1467] Processing CreateTablet for tablet e3783fc7fe434696898ba14ab77a9058 (DEFAULT_TABLE table=default.Table_953eb3a1426a4a479cb256cc68e6d4be [id=df45c16e38c84c40bea02ce036be7b4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.359163 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193ec0b463d04835b74df80229c85cde" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.359476 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3783fc7fe434696898ba14ab77a9058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.359455 6553 raft_consensus.cc:2781] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.359592 6553 raft_consensus.cc:686] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:05.359751 5654 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.359840 6553 consensus_queue.cc:237] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.361032 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3783fc7fe434696898ba14ab77a9058. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.363914 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fbe0f58bad541e7bcfb611105328d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.364396 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9846e0eab74388af231e9942a6881a. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:05.374786 5656 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 193ec0b463d04835b74df80229c85cde
I20240507 08:12:05.374919 5656 leader_election.cc:302] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.375144 6675 raft_consensus.cc:2726] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.376641 5840 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.376775 5840 ts_tablet_manager.cc:1367] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.070s user 0.023s sys 0.032s
I20240507 08:12:05.377889 6676 raft_consensus.cc:2726] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.377810 5840 raft_consensus.cc:348] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.378000 5840 raft_consensus.cc:374] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.378052 5840 raft_consensus.cc:729] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.378268 5840 consensus_queue.cc:260] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.380190 5840 ts_tablet_manager.cc:1398] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:05.380496 5840 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.381865 6676 raft_consensus.cc:482] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.382035 6676 raft_consensus.cc:504] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.382658 6676 leader_election.cc:288] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.383083 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.383188 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.383271 5720 raft_consensus.cc:2454] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:12:05.383848 5378 leader_election.cc:341] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.386565 5377 leader_election.cc:302] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.386881 6684 raft_consensus.cc:2781] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.386981 6684 raft_consensus.cc:482] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.387064 6684 raft_consensus.cc:3037] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.393421 6603 raft_consensus.cc:482] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.393610 6603 raft_consensus.cc:504] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.394317 6603 leader_election.cc:288] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:05.395262 5520 leader_election.cc:341] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: fde460fdcf254af6a3cdaaac22ac2d93
W20240507 08:12:05.396145 5517 leader_election.cc:341] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: fde460fdcf254af6a3cdaaac22ac2d93
I20240507 08:12:05.396275 5517 leader_election.cc:302] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.396656 6603 raft_consensus.cc:2726] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.397672 5839 tablet_bootstrap.cc:492] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.397823 5839 ts_tablet_manager.cc:1367] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20240507 08:12:05.398706 5839 raft_consensus.cc:348] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.398910 5839 raft_consensus.cc:374] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.398994 5839 raft_consensus.cc:729] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.401427 5839 consensus_queue.cc:260] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.402606 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_558ec8354d9943f6ad3e2be4108aefc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.403168 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_558ec8354d9943f6ad3e2be4108aefc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.406894 5839 ts_tablet_manager.cc:1398] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:05.407167 5839 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:12:05.410146 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6def4bf6a7554a4090a39b1823c4097c consensus state has no leader
I20240507 08:12:05.410519 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3b48d6ff43424a76a86e1880f32c59f7" } new_table_name: "default.Table_510d804e03f54c96b5321a44a0b1dd38" modify_external_catalogs: true
I20240507 08:12:05.411844 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 93c68ddcfa7c40c180b6b12fabe165d6
I20240507 08:12:05.412266 5843 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.412582 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2aaf5f336efc491a8eacf261247f65bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.413060 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aaf5f336efc491a8eacf261247f65bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:05.413139 5654 consensus_peers.cc:485] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.413863 5843 tablet_bootstrap.cc:654] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.414736 5839 tablet_bootstrap.cc:654] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.415472 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.418238 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 916e33e40ef44d06a359b82d8fe48157 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.418646 5840 tablet_bootstrap.cc:654] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:05.419449 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4dc71de9f20547c8a62d320b11b4c0f9 has no consensus state
I20240507 08:12:05.419734 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.423131 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.423594 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 93c68ddcfa7c40c180b6b12fabe165d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.423970 6684 raft_consensus.cc:504] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.425405 6684 leader_election.cc:288] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.425541 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:05.425768 5720 raft_consensus.cc:3037] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.431221 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 708) took 1135 ms. Trace:
I20240507 08:12:05.431309 6513 rpcz_store.cc:276] 0507 08:12:04.295743 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.295796 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:05.168662 (+872866us) ts_tablet_manager.cc:1120] Deleting tablet a7df45c39d1b490db97c72a0eb6cf3ba
0507 08:12:05.168671 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.245899 (+ 77228us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.337756 (+ 91857us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.431199 (+ 93443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":872689}
I20240507 08:12:05.431658 6513 tablet_replica.cc:330] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:05.431828 6513 raft_consensus.cc:2229] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:05.432045 6513 pending_rounds.cc:62] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:05.432103 6513 pending_rounds.cc:68] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924771815186432 op_type: NO_OP noop_request { }
I20240507 08:12:05.432193 6513 raft_consensus.cc:2866] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:05.432256 6513 raft_consensus.cc:2258] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.433056 6513 ts_tablet_manager.cc:1875] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.433826 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98aa4ec7b93943e5bf8f9758b3bb71e0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.434355 5378 leader_election.cc:341] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.435076 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 724) took 1038 ms. Trace:
I20240507 08:12:05.435173 5841 rpcz_store.cc:276] 0507 08:12:04.396879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.396942 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:05.139455 (+742513us) ts_tablet_manager.cc:1120] Deleting tablet d45d3fbbe3764261a3a24523a77c6443
0507 08:12:05.139464 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.232520 (+ 93056us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.333464 (+100944us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.435036 (+101572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":742396}
I20240507 08:12:05.435523 5841 tablet_replica.cc:330] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:05.435704 5841 raft_consensus.cc:2229] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.435837 5841 raft_consensus.cc:2258] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.436327 5841 ts_tablet_manager.cc:1875] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.437785 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d45d3fbbe3764261a3a24523a77c6443 (table default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]) successfully deleted
I20240507 08:12:05.437892 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet a7df45c39d1b490db97c72a0eb6cf3ba (table default.Table_0fafe65af5f34be4b333cb6baa327f94 [id=58282ffddcba4e4db7736a325a4a530a]) successfully deleted
I20240507 08:12:05.440521 5840 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.440641 5840 ts_tablet_manager.cc:1367] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.000s
I20240507 08:12:05.441298 5840 raft_consensus.cc:348] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.441470 5840 raft_consensus.cc:374] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.441511 5840 raft_consensus.cc:729] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.441907 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b9d330db1b874cb487e83379f57a55e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.442523 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9d330db1b874cb487e83379f57a55e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.442787 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet b22334378eec4ab3885d0f9d2d812a3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.445147 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b28dbe67129643cfad8ac14f10aa72c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.446064 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b28dbe67129643cfad8ac14f10aa72c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.447876 5840 consensus_queue.cc:260] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.448460 5840 ts_tablet_manager.cc:1398] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:05.448743 5840 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.450769 5840 tablet_bootstrap.cc:654] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.456562 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 689) took 1228 ms. Trace:
I20240507 08:12:05.456702 6547 rpcz_store.cc:276] 0507 08:12:04.227853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.227910 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:04.927851 (+699941us) ts_tablet_manager.cc:1120] Deleting tablet f5e48a2b4a0e4088ad009a7f8fe77f3f
0507 08:12:04.927859 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.134461 (+206602us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.334719 (+200258us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.456513 (+121794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":699709}
I20240507 08:12:05.457386 6547 tablet_replica.cc:330] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:05.457613 6547 raft_consensus.cc:2229] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.457787 6547 raft_consensus.cc:2258] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.458278 6547 ts_tablet_manager.cc:1875] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.462163 5840 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.462278 5840 ts_tablet_manager.cc:1367] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240507 08:12:05.462913 5840 raft_consensus.cc:348] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.463078 5840 raft_consensus.cc:374] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.463135 5840 raft_consensus.cc:729] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.463449 5840 consensus_queue.cc:260] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.463935 5840 ts_tablet_manager.cc:1398] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:05.464231 5840 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.464587 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f5e48a2b4a0e4088ad009a7f8fe77f3f (table default.Table_1bdf5a587592440cb76f8cb1b5e0cd5c [id=feeaf98e940f4006814716f3df172e51]) successfully deleted
I20240507 08:12:05.466019 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d451d03349794726a71c255a7f177436
I20240507 08:12:05.466480 5257 catalog_manager.cc:6554] ReplaceTablet: tablet d451d03349794726a71c255a7f177436 deleted and replaced by tablet 1227b63fbc6b4625bef0b37404b76a6f
W20240507 08:12:05.468237 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:05.468789 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d03349794726a71c255a7f177436 with delete_type TABLET_DATA_DELETED (replaced by tablet 1227b63fbc6b4625bef0b37404b76a6f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.466884 5840 tablet_bootstrap.cc:654] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.470103 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d03349794726a71c255a7f177436 with delete_type TABLET_DATA_DELETED (replaced by tablet 1227b63fbc6b4625bef0b37404b76a6f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.470810 5443 raft_consensus.cc:1261] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Refusing update from remote peer 1cbb887175f345ada2433adaec906fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:12:05.471347 6675 consensus_queue.cc:1035] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:05.473018 5654 consensus_peers.cc:485] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.478917 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d03349794726a71c255a7f177436 with delete_type TABLET_DATA_DELETED (replaced by tablet 1227b63fbc6b4625bef0b37404b76a6f) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:05.483965 5378 consensus_peers.cc:485] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.493016 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_881b71c36b4048b5acf1ab063ccbf33b" } new_table_name: "default.Table_3c1d67dd81bd47a0b00488a63e68d269" modify_external_catalogs: true
I20240507 08:12:05.497826 5840 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.497972 5840 ts_tablet_manager.cc:1367] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20240507 08:12:05.498111 5843 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.498309 5843 ts_tablet_manager.cc:1367] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.086s user 0.015s sys 0.002s
I20240507 08:12:05.498780 5840 raft_consensus.cc:348] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.498996 5840 raft_consensus.cc:374] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.499063 5840 raft_consensus.cc:729] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.499222 5843 raft_consensus.cc:348] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.499284 5840 consensus_queue.cc:260] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.499821 5840 ts_tablet_manager.cc:1398] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:05.500180 5840 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.501642 5843 raft_consensus.cc:374] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.501688 5840 tablet_bootstrap.cc:654] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.501824 5843 raft_consensus.cc:729] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.502146 5843 consensus_queue.cc:260] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.503276 5839 tablet_bootstrap.cc:492] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.503440 5839 ts_tablet_manager.cc:1367] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.000s
I20240507 08:12:05.503863 5843 ts_tablet_manager.cc:1398] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:05.504796 5839 raft_consensus.cc:348] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.505017 5839 raft_consensus.cc:374] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.505081 5839 raft_consensus.cc:729] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.506507 5843 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.506312 5839 consensus_queue.cc:260] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.510629 5843 tablet_bootstrap.cc:654] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.511615 6603 raft_consensus.cc:482] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.511750 6603 raft_consensus.cc:504] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.512341 6603 leader_election.cc:288] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:05.513748 5517 leader_election.cc:341] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5f2b43daab6b4deeac94fc6f1ff34a76
W20240507 08:12:05.514159 5520 leader_election.cc:341] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5f2b43daab6b4deeac94fc6f1ff34a76
I20240507 08:12:05.514271 5520 leader_election.cc:302] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.514991 6603 raft_consensus.cc:2726] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.518659 5840 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.518797 5840 ts_tablet_manager.cc:1367] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.007s
I20240507 08:12:05.519460 5840 raft_consensus.cc:348] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.519660 5840 raft_consensus.cc:374] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.519719 5840 raft_consensus.cc:729] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.519896 5840 consensus_queue.cc:260] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.520399 5840 ts_tablet_manager.cc:1398] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:05.520622 5840 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.522107 6603 raft_consensus.cc:482] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.522284 6603 raft_consensus.cc:504] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.522969 6603 leader_election.cc:288] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.523658 5840 tablet_bootstrap.cc:654] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.524083 5843 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.524281 5843 ts_tablet_manager.cc:1367] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
W20240507 08:12:05.524456 5520 leader_election.cc:341] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 88a2cfb5f046434c8340234a447a7e65
I20240507 08:12:05.524938 5843 raft_consensus.cc:348] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.525131 5843 raft_consensus.cc:374] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.525190 5843 raft_consensus.cc:729] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.525408 5843 consensus_queue.cc:260] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.525902 5843 ts_tablet_manager.cc:1398] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:05.526168 5843 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.527554 5843 tablet_bootstrap.cc:654] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.530642 5720 raft_consensus.cc:2454] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:12:05.531657 5517 leader_election.cc:341] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 88a2cfb5f046434c8340234a447a7e65
I20240507 08:12:05.531788 5517 leader_election.cc:302] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.533164 6603 raft_consensus.cc:2726] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.533463 5377 leader_election.cc:302] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.533957 6684 raft_consensus.cc:2781] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.534099 6684 raft_consensus.cc:686] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:05.534346 6684 consensus_queue.cc:237] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.541203 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2b13fbddcab5460db69f43e63f490fba
I20240507 08:12:05.541836 5264 catalog_manager.cc:6554] ReplaceTablet: tablet 2b13fbddcab5460db69f43e63f490fba deleted and replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f
I20240507 08:12:05.542166 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b13fbddcab5460db69f43e63f490fba with delete_type TABLET_DATA_DELETED (replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.545087 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 5782cfa5eca74c19a2e724a9124597f2 (table default.Table_ee5279fc19a046aebae1f622389f59a2 [id=c6a13f9902b0424ca8df2184779a407c]). Expected version 3 got 0
I20240507 08:12:05.547688 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b13fbddcab5460db69f43e63f490fba with delete_type TABLET_DATA_DELETED (replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.545773 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2e80ae4bf74945cc82095e3a11e1939b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.548522 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e80ae4bf74945cc82095e3a11e1939b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.546476 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9178d27018a341e2bc0fb33f4e36c267"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.550148 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9178d27018a341e2bc0fb33f4e36c267 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.547050 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet c9e2deaadfe34c2dbdc9819729cb4890 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.552783 6614 consensus_queue.cc:1035] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:05.555756 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b13fbddcab5460db69f43e63f490fba with delete_type TABLET_DATA_DELETED (replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.556918 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bcd651b469bc460c8df842333fc5f367" } new_table_name: "default.Table_4ab977deaf2b47b281a5d9ae655f0f43" modify_external_catalogs: true
W20240507 08:12:05.559636 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 3dc49fd6841c420288aa1df3950a088b (table default.Table_687fbbf6bfdc4aab91a289e90561fafd [id=0beae72e42824f1e969a306a8d5658d5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3dc49fd6841c420288aa1df3950a088b consensus state has no leader
I20240507 08:12:05.560647 6733 tablet.cc:1722] T 5782cfa5eca74c19a2e724a9124597f2 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:05.570622 5843 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.570771 5843 ts_tablet_manager.cc:1367] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.006s
I20240507 08:12:05.571569 5843 raft_consensus.cc:348] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.571759 5843 raft_consensus.cc:374] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.571846 5843 raft_consensus.cc:729] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.572041 5843 consensus_queue.cc:260] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.572981 5843 ts_tablet_manager.cc:1398] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:05.573263 5843 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.575484 5843 tablet_bootstrap.cc:654] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.575301 6734 tablet.cc:1722] T 5782cfa5eca74c19a2e724a9124597f2 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:05.576421 5840 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.576555 5840 ts_tablet_manager.cc:1367] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.056s user 0.012s sys 0.008s
I20240507 08:12:05.577486 5840 raft_consensus.cc:348] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.577750 5840 raft_consensus.cc:374] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.577821 5840 raft_consensus.cc:729] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.579478 5839 ts_tablet_manager.cc:1398] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.076s user 0.001s sys 0.001s
W20240507 08:12:05.585135 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90dcd7eee62b458ea4661a05a1bfcac4 has no consensus state
I20240507 08:12:05.587760 5839 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.589490 5839 tablet_bootstrap.cc:654] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.590864 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.591104 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.591913 5843 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.592133 5843 ts_tablet_manager.cc:1367] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20240507 08:12:05.592844 5843 raft_consensus.cc:348] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.593122 5843 raft_consensus.cc:374] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.593231 5843 raft_consensus.cc:729] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.593410 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_48b51787d9ce4ff1b9b8744ed86885bb" } modify_external_catalogs: true
I20240507 08:12:05.593525 5843 consensus_queue.cc:260] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.593686 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_48b51787d9ce4ff1b9b8744ed86885bb" } modify_external_catalogs: true
I20240507 08:12:05.594399 5843 ts_tablet_manager.cc:1398] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:05.600122 5840 consensus_queue.cc:260] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.600826 5840 ts_tablet_manager.cc:1398] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.024s user 0.016s sys 0.000s
I20240507 08:12:05.602057 5843 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.604050 5843 tablet_bootstrap.cc:654] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.606331 5840 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.607916 5840 tablet_bootstrap.cc:654] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.607941 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bf6b95a29b9d466eabf2276567fe5fa9" } modify_external_catalogs: true
I20240507 08:12:05.608144 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bf6b95a29b9d466eabf2276567fe5fa9" } modify_external_catalogs: true
I20240507 08:12:05.611809 5843 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.611966 5843 ts_tablet_manager.cc:1367] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.001s
I20240507 08:12:05.612682 5843 raft_consensus.cc:348] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.612900 5843 raft_consensus.cc:374] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.612987 5843 raft_consensus.cc:729] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.613209 5843 consensus_queue.cc:260] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.614789 6614 raft_consensus.cc:482] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.614939 6614 raft_consensus.cc:504] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.615556 6614 leader_election.cc:288] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.615979 5843 ts_tablet_manager.cc:1398] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:12:05.616264 5843 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.616317 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.616436 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:05.616951 5377 leader_election.cc:341] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:05.617491 5378 leader_election.cc:341] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.617658 5378 leader_election.cc:302] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.618278 6676 raft_consensus.cc:2726] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.618840 5843 tablet_bootstrap.cc:654] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.630131 5841 ts_tablet_manager.cc:1888] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.630317 5841 log.cc:1196] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7027b4cefa4d41659f819ee4326470c4
I20240507 08:12:05.630501 5443 raft_consensus.cc:1261] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Refusing update from remote peer 1cbb887175f345ada2433adaec906fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:12:05.630666 5841 ts_tablet_manager.cc:1909] T 7027b4cefa4d41659f819ee4326470c4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:05.631075 6547 ts_tablet_manager.cc:1888] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.631183 6547 log.cc:1196] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/072702ef2a6d43248132da6aa3a2d493
I20240507 08:12:05.631204 6675 consensus_queue.cc:1035] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:05.631453 6547 ts_tablet_manager.cc:1909] T 072702ef2a6d43248132da6aa3a2d493 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:05.634295 6603 raft_consensus.cc:482] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.634446 6603 raft_consensus.cc:504] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.635128 6603 leader_election.cc:288] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:05.637542 5517 leader_election.cc:341] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 916e33e40ef44d06a359b82d8fe48157
W20240507 08:12:05.637920 5654 consensus_peers.cc:485] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.640544 6513 ts_tablet_manager.cc:1888] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:05.640671 6513 log.cc:1196] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5089db17ca904a04878d83209001c54e
I20240507 08:12:05.641074 6513 ts_tablet_manager.cc:1909] T 5089db17ca904a04878d83209001c54e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:05.641158 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71341b71104946439f1c74a9dc7b9c3d" } modify_external_catalogs: true
I20240507 08:12:05.641373 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_71341b71104946439f1c74a9dc7b9c3d" } modify_external_catalogs: true
W20240507 08:12:05.645959 5520 leader_election.cc:341] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 916e33e40ef44d06a359b82d8fe48157
I20240507 08:12:05.646080 5520 leader_election.cc:302] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.646551 6603 raft_consensus.cc:2726] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:05.652967 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124ede6467314270a2d99e0e45097c52 consensus state has no leader
I20240507 08:12:05.655269 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_682aa156946f429890d5095fa23f097e" } modify_external_catalogs: true
I20240507 08:12:05.655452 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_682aa156946f429890d5095fa23f097e" } modify_external_catalogs: true
I20240507 08:12:05.666568 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c795df0998644b1f933ec20bdb5a2d84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.667080 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c795df0998644b1f933ec20bdb5a2d84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.674945 5840 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.675102 5840 ts_tablet_manager.cc:1367] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.069s user 0.001s sys 0.005s
I20240507 08:12:05.675891 5840 raft_consensus.cc:348] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.676074 5840 raft_consensus.cc:374] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.676121 5840 raft_consensus.cc:729] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.677148 5840 consensus_queue.cc:260] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.677758 5840 ts_tablet_manager.cc:1398] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:05.678251 6508 raft_consensus.cc:482] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.678380 6508 raft_consensus.cc:504] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.678975 6508 leader_election.cc:288] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.679394 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dea0e2b1b090454cbd15a00f7e792d6e" } modify_external_catalogs: true
I20240507 08:12:05.679595 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dea0e2b1b090454cbd15a00f7e792d6e" } modify_external_catalogs: true
I20240507 08:12:05.680354 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.681418 5654 leader_election.cc:341] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.681557 5840 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:05.681771 5656 consensus_peers.cc:485] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa14901cdf14b72bea365b04ad521bd. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.682075 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c9e2deaadfe34c2dbdc9819729cb4890
I20240507 08:12:05.682353 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.682559 5258 catalog_manager.cc:6554] ReplaceTablet: tablet c9e2deaadfe34c2dbdc9819729cb4890 deleted and replaced by tablet 8c88421118434749abf4126b08b115e5
I20240507 08:12:05.682556 5443 raft_consensus.cc:2454] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.682876 5656 leader_election.cc:302] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.682897 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c96ab339396440379b168db4805f8779" } new_table_name: "default.Table_63c3ddcab80f42fdb70f3161d8639842" modify_external_catalogs: true
I20240507 08:12:05.683211 6508 raft_consensus.cc:2781] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.683280 6508 raft_consensus.cc:482] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.683328 5840 tablet_bootstrap.cc:654] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.683346 6508 raft_consensus.cc:3037] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:05.684252 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ddcb720c09c5486995391837fc54e7ce consensus state has no leader
I20240507 08:12:05.685112 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_34e4975648c24bb59e66757e3319a0a0" } new_table_name: "default.Table_3c747008c670486b8d63c755d6ca129b" modify_external_catalogs: true
I20240507 08:12:05.686254 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c1c6b488b9014841899597d858bda5e2
I20240507 08:12:05.686734 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet aca8c38841db45e38ae5d8d965a16d0e from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.687266 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 077ccc3946b34008bea7b80d8aee2662
I20240507 08:12:05.688704 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c88421118434749abf4126b08b115e5) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.689810 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c6b488b9014841899597d858bda5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.690058 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 077ccc3946b34008bea7b80d8aee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.691357 5839 tablet_bootstrap.cc:492] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.691531 5839 ts_tablet_manager.cc:1367] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.104s user 0.005s sys 0.001s
I20240507 08:12:05.691641 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c88421118434749abf4126b08b115e5) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.691857 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c6b488b9014841899597d858bda5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.692080 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 077ccc3946b34008bea7b80d8aee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.692775 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e2deaadfe34c2dbdc9819729cb4890 with delete_type TABLET_DATA_DELETED (replaced by tablet 8c88421118434749abf4126b08b115e5) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.692919 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 077ccc3946b34008bea7b80d8aee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.693284 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet c1c6b488b9014841899597d858bda5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.693347 5839 raft_consensus.cc:348] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.693611 5839 raft_consensus.cc:374] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.693696 5839 raft_consensus.cc:729] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.693717 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_398e8d420fb14944ac6abf845df55580"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.693950 5839 consensus_queue.cc:260] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:05.694586 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_398e8d420fb14944ac6abf845df55580 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.695554 6743 tablet.cc:1722] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:05.696772 6744 tablet.cc:1722] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:05.701536 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 5554ebc621c24d459668f2cefc20b641 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:51 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.702144 5839 ts_tablet_manager.cc:1398] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20240507 08:12:05.709332 5839 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.709497 5843 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.709681 5843 ts_tablet_manager.cc:1367] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.093s user 0.021s sys 0.037s
I20240507 08:12:05.712275 5839 tablet_bootstrap.cc:654] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.732041 6748 tablet.cc:1722] T 169626f9a0bc4c6f9d8be11c0a802816 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:05.733726 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 706) took 1438 ms. Trace:
I20240507 08:12:05.733827 6513 rpcz_store.cc:276] 0507 08:12:04.295575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.295901 (+ 326us) service_pool.cc:225] Handling call
0507 08:12:05.431613 (+1135712us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:12:05.431622 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.533525 (+101903us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.640455 (+106930us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.733702 (+ 93247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1135506}
I20240507 08:12:05.734135 6513 tablet_replica.cc:330] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:05.734303 6513 raft_consensus.cc:2229] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.734423 6513 raft_consensus.cc:2258] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.734922 6513 ts_tablet_manager.cc:1875] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.734931 5843 raft_consensus.cc:348] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.735198 5843 raft_consensus.cc:374] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.735297 5843 raft_consensus.cc:729] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.735599 5843 consensus_queue.cc:260] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.735801 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 721) took 1339 ms. Trace:
I20240507 08:12:05.735882 5841 rpcz_store.cc:276] 0507 08:12:04.396554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.396796 (+ 242us) service_pool.cc:225] Handling call
0507 08:12:05.435466 (+1038670us) ts_tablet_manager.cc:1120] Deleting tablet 7027b4cefa4d41659f819ee4326470c4
0507 08:12:05.435475 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.537205 (+101730us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.629954 (+ 92749us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.735780 (+105826us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":593,"tablet-delete.queue_time_us":1038560}
I20240507 08:12:05.736286 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 728) took 1339 ms. Trace:
I20240507 08:12:05.736382 5841 rpcz_store.cc:276] 0507 08:12:04.397241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.397294 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:05.736162 (+1338868us) ts_tablet_manager.cc:1120] Deleting tablet 53bc9bf1c04e4015a152cb750184c9ce
0507 08:12:05.736172 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.736271 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":53,"tablet-delete.queue_time_us":1338647}
I20240507 08:12:05.736835 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 690) took 1472 ms. Trace:
I20240507 08:12:05.736943 6547 rpcz_store.cc:276] 0507 08:12:04.264721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.264770 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:05.457316 (+1192546us) ts_tablet_manager.cc:1120] Deleting tablet 072702ef2a6d43248132da6aa3a2d493
0507 08:12:05.457325 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.540007 (+ 82682us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.630960 (+ 90953us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.736785 (+105825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1192356}
I20240507 08:12:05.737354 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 692) took 1398 ms. Trace:
I20240507 08:12:05.737424 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 072702ef2a6d43248132da6aa3a2d493 (table default.Table_c6bf901b2c4a491e8aa4c357670a5d08 [id=3d99e1ae7dc34ac49843ce5f159f891e]) successfully deleted
I20240507 08:12:05.737457 6547 rpcz_store.cc:276] 0507 08:12:04.339180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.339240 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:05.737211 (+1397971us) ts_tablet_manager.cc:1120] Deleting tablet 07a9324570984372af7b9574f0cf455a
0507 08:12:05.737220 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.737304 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1397784}
I20240507 08:12:05.737814 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 693) took 1387 ms. Trace:
W20240507 08:12:05.737886 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 07a9324570984372af7b9574f0cf455a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9324570984372af7b9574f0cf455a already in progress: creating tablet
I20240507 08:12:05.737900 6547 rpcz_store.cc:276] 0507 08:12:04.350434 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.350493 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:05.737695 (+1387202us) ts_tablet_manager.cc:1120] Deleting tablet 85dc58133d2b4f5496b4e570ae6494c0
0507 08:12:05.737703 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.737774 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1387000}
I20240507 08:12:05.738209 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 694) took 1387 ms. Trace:
I20240507 08:12:05.738296 6547 rpcz_store.cc:276] 0507 08:12:04.350954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.351001 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:05.738098 (+1387097us) ts_tablet_manager.cc:1120] Deleting tablet 8e1f91718260466d99acfc9892dea536
0507 08:12:05.738105 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.738168 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386939}
I20240507 08:12:05.738615 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 695) took 1387 ms. Trace:
I20240507 08:12:05.738706 6547 rpcz_store.cc:276] 0507 08:12:04.351398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.351442 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:05.738490 (+1387048us) ts_tablet_manager.cc:1120] Deleting tablet 039dcaf121b94ebab94ec522346d9d82
0507 08:12:05.738498 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.738561 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386915}
I20240507 08:12:05.739005 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 696) took 1360 ms. Trace:
I20240507 08:12:05.739099 6547 rpcz_store.cc:276] 0507 08:12:04.378289 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.378349 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:05.738888 (+1360539us) ts_tablet_manager.cc:1120] Deleting tablet b5d4993c287a40cf964877b5464372a6
0507 08:12:05.738895 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.738962 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1360352}
I20240507 08:12:05.739421 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 700) took 1342 ms. Trace:
I20240507 08:12:05.739521 6547 rpcz_store.cc:276] 0507 08:12:04.396514 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.396555 (+ 41us) service_pool.cc:225] Handling call
0507 08:12:05.739290 (+1342735us) ts_tablet_manager.cc:1120] Deleting tablet 0c48c9720c1a4a8b849730f28f6e00d9
0507 08:12:05.739300 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.739369 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1342532}
W20240507 08:12:05.739820 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 85dc58133d2b4f5496b4e570ae6494c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85dc58133d2b4f5496b4e570ae6494c0 already in progress: creating tablet
I20240507 08:12:05.739854 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 699) took 1343 ms. Trace:
I20240507 08:12:05.739933 6547 rpcz_store.cc:276] 0507 08:12:04.396420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.397437 (+ 1017us) service_pool.cc:225] Handling call
0507 08:12:05.739741 (+1342304us) ts_tablet_manager.cc:1120] Deleting tablet 53bc9bf1c04e4015a152cb750184c9ce
0507 08:12:05.739748 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.739817 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1342101}
W20240507 08:12:05.740048 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8e1f91718260466d99acfc9892dea536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e1f91718260466d99acfc9892dea536 already in progress: creating tablet
W20240507 08:12:05.740276 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 039dcaf121b94ebab94ec522346d9d82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 039dcaf121b94ebab94ec522346d9d82 already in progress: creating tablet
W20240507 08:12:05.740562 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b5d4993c287a40cf964877b5464372a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d4993c287a40cf964877b5464372a6 already in progress: creating tablet
I20240507 08:12:05.740882 6734 tablet.cc:1722] T 169626f9a0bc4c6f9d8be11c0a802816 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:05.743279 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5089db17ca904a04878d83209001c54e (table default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]) successfully deleted
I20240507 08:12:05.743675 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 707) took 1210 ms. Trace:
I20240507 08:12:05.743789 6547 rpcz_store.cc:276] 0507 08:12:04.532815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.533003 (+ 188us) service_pool.cc:225] Handling call
0507 08:12:05.740126 (+1207123us) ts_tablet_manager.cc:1120] Deleting tablet 3b16a81588394d28945331e233e1f1db
0507 08:12:05.740134 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.743641 (+ 3507us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206898}
I20240507 08:12:05.744150 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7027b4cefa4d41659f819ee4326470c4 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]) successfully deleted
I20240507 08:12:05.744163 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 713) took 1020 ms. Trace:
I20240507 08:12:05.744264 6547 rpcz_store.cc:276] 0507 08:12:04.723702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.725533 (+ 1831us) service_pool.cc:225] Handling call
0507 08:12:05.744023 (+1018490us) ts_tablet_manager.cc:1120] Deleting tablet 4716c1285e91432a92b25ab13ada2779
0507 08:12:05.744031 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.744123 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1018254}
W20240507 08:12:05.744400 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 53bc9bf1c04e4015a152cb750184c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bc9bf1c04e4015a152cb750184c9ce already in progress: creating tablet
I20240507 08:12:05.745374 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 712) took 1021 ms. Trace:
I20240507 08:12:05.745471 6547 rpcz_store.cc:276] 0507 08:12:04.723546 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.761020 (+ 37474us) service_pool.cc:225] Handling call
0507 08:12:05.745253 (+984233us) ts_tablet_manager.cc:1120] Deleting tablet 279e95d9804c44eaa2c23a4ec02c84ff
0507 08:12:05.745261 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.745331 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":983875}
I20240507 08:12:05.746191 6547 tablet_replica.cc:330] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:05.746387 6547 raft_consensus.cc:2229] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.746560 6547 raft_consensus.cc:2258] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.747119 6547 ts_tablet_manager.cc:1875] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.748108 6508 raft_consensus.cc:504] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.748690 6508 leader_election.cc:288] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.748879 5841 tablet_replica.cc:330] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:05.749056 5841 raft_consensus.cc:2229] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:05.749177 5841 raft_consensus.cc:2258] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:05.749653 5841 ts_tablet_manager.cc:1875] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:05.750255 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet c43f3de489904f2ba2a637d408619bfc from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:05.751827 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0c48c9720c1a4a8b849730f28f6e00d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c48c9720c1a4a8b849730f28f6e00d9 already in progress: creating tablet
W20240507 08:12:05.752111 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 53bc9bf1c04e4015a152cb750184c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bc9bf1c04e4015a152cb750184c9ce already in progress: creating tablet
W20240507 08:12:05.752338 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3b16a81588394d28945331e233e1f1db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b16a81588394d28945331e233e1f1db already in progress: creating tablet
I20240507 08:12:05.752506 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.752591 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4716c1285e91432a92b25ab13ada2779 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4716c1285e91432a92b25ab13ada2779 already in progress: creating tablet
W20240507 08:12:05.752784 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 279e95d9804c44eaa2c23a4ec02c84ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279e95d9804c44eaa2c23a4ec02c84ff already in progress: creating tablet
W20240507 08:12:05.752957 5654 leader_election.cc:341] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.736294 5843 ts_tablet_manager.cc:1398] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 08:12:05.753455 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 79524df2415146ab926425c42068626c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79524df2415146ab926425c42068626c already in progress: creating tablet
I20240507 08:12:05.753679 5843 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:05.753731 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 33875331045146a9a02559a0b37d5c6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33875331045146a9a02559a0b37d5c6f already in progress: creating tablet
W20240507 08:12:05.753945 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 63685e9ef255477db589fa572b087b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63685e9ef255477db589fa572b087b57 already in progress: creating tablet
I20240507 08:12:05.754989 5843 tablet_bootstrap.cc:654] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.755142 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e447562e82347a584e39586500c27c2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.755360 5443 raft_consensus.cc:3037] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.756435 5840 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.756567 5840 ts_tablet_manager.cc:1367] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.002s
I20240507 08:12:05.757200 5840 raft_consensus.cc:348] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.757392 5840 raft_consensus.cc:374] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.757447 5840 raft_consensus.cc:729] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.757678 5840 consensus_queue.cc:260] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.758293 5840 ts_tablet_manager.cc:1398] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:05.758553 5840 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.759779 5840 tablet_bootstrap.cc:654] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.759642 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_215f0b92297748bd926f9b40fea2f149"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.761255 5839 tablet_bootstrap.cc:492] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.761399 5839 ts_tablet_manager.cc:1367] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.000s
I20240507 08:12:05.761523 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_801bce464efb4d0781fea9659af5f26c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.762112 5839 raft_consensus.cc:348] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.762319 5839 raft_consensus.cc:374] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.762382 5839 raft_consensus.cc:729] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.765846 6508 raft_consensus.cc:482] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.765969 6508 raft_consensus.cc:504] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.766518 6508 leader_election.cc:288] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.767889 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45d3fbbe3764261a3a24523a77c6443" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.768301 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_215f0b92297748bd926f9b40fea2f149 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:05.768435 5654 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:05.768340 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e17502bb4ac64f6ca9c77a3b00ab9f4a consensus state has no leader
I20240507 08:12:05.768952 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_98d20b30912d46dd9059405387dde59a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.769455 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98d20b30912d46dd9059405387dde59a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:05.769609 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_801bce464efb4d0781fea9659af5f26c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.770493 5840 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.770627 5840 ts_tablet_manager.cc:1367] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
W20240507 08:12:05.771253 5656 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:05.771363 5656 leader_election.cc:302] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.771756 6508 raft_consensus.cc:2726] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.771338 5840 raft_consensus.cc:348] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.772094 5840 raft_consensus.cc:374] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.772176 5840 raft_consensus.cc:729] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.772367 5840 consensus_queue.cc:260] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:05.777068 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f748a59b4a347e288d534ef1296d72c has no consensus state
I20240507 08:12:05.781867 5839 consensus_queue.cc:260] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.783206 5840 ts_tablet_manager.cc:1398] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240507 08:12:05.785082 5840 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.785969 5839 ts_tablet_manager.cc:1398] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240507 08:12:05.786681 5839 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.788673 5839 tablet_bootstrap.cc:654] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.793306 6614 raft_consensus.cc:482] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.793460 6614 raft_consensus.cc:504] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.794242 6614 leader_election.cc:288] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.794826 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c978ec8bca4aa5b2ffaf22c3290747" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.795015 5720 raft_consensus.cc:2454] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:05.795547 5377 leader_election.cc:302] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:05.796242 6614 raft_consensus.cc:2781] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.796320 6614 raft_consensus.cc:482] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.796386 6614 raft_consensus.cc:3037] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.796286 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_44cd8bf73a544ecf89f263225686b5fd" } new_table_name: "default.Table_bc3afc777e354e2e8a8ae7266a761d73" modify_external_catalogs: true
I20240507 08:12:05.797757 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 34bebec088a34ebbb06d2e707d54da4c with delete_type TABLET_DATA_DELETED (replaced by tablet 957775c01be94cc081db344c95a81310) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.798646 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 3618b416194f4effb29b823709593359 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:05.798703 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c978ec8bca4aa5b2ffaf22c3290747" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.798923 5840 tablet_bootstrap.cc:654] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:05.799293 5378 leader_election.cc:341] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.801092 6508 raft_consensus.cc:482] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.801234 6508 raft_consensus.cc:504] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.801872 6508 leader_election.cc:288] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.803383 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.803956 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.804159 5444 raft_consensus.cc:2454] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:05.804798 5654 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.804699 5257 catalog_manager.cc:5484] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:05.805215 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
I20240507 08:12:05.805204 5656 leader_election.cc:302] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.805601 6508 raft_consensus.cc:2781] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.805671 6508 raft_consensus.cc:482] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.805725 6508 raft_consensus.cc:3037] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.806587 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_15c0a7e5faba41c6a569c92d6eb7d32d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.807124 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15c0a7e5faba41c6a569c92d6eb7d32d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.810397 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab43b817cb448259b9c939ede3bcb6d (DEFAULT_TABLE table=default.Table_0bece17329174cf886f812c040ec88df [id=aeab7c65285a4de983a2612e9eb48fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.811801 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 05ed335aadfe432cac42539e196f095a (DEFAULT_TABLE table=default.Table_f5289d2de1e64222b8a1faf354cf0048 [id=24c138ae9237417280d0904ee7ecab00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.812332 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab43b817cb448259b9c939ede3bcb6d (DEFAULT_TABLE table=default.Table_0bece17329174cf886f812c040ec88df [id=aeab7c65285a4de983a2612e9eb48fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.812816 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 8ab43b817cb448259b9c939ede3bcb6d (DEFAULT_TABLE table=default.Table_0bece17329174cf886f812c040ec88df [id=aeab7c65285a4de983a2612e9eb48fd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.814168 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 05ed335aadfe432cac42539e196f095a (DEFAULT_TABLE table=default.Table_f5289d2de1e64222b8a1faf354cf0048 [id=24c138ae9237417280d0904ee7ecab00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.814371 5843 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.814504 5843 ts_tablet_manager.cc:1367] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.003s
I20240507 08:12:05.815240 5843 raft_consensus.cc:348] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.815479 5843 raft_consensus.cc:374] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.815526 5843 raft_consensus.cc:729] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.815779 5843 consensus_queue.cc:260] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.816470 6675 raft_consensus.cc:482] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.816591 6675 raft_consensus.cc:504] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.817020 5843 ts_tablet_manager.cc:1398] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:05.817179 6675 leader_election.cc:288] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:05.817281 5843 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.818856 5843 tablet_bootstrap.cc:654] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.818289 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.819008 5444 raft_consensus.cc:2454] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.819818 5656 leader_election.cc:302] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:05.820327 6553 raft_consensus.cc:2781] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.820408 6553 raft_consensus.cc:482] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.820479 6553 raft_consensus.cc:3037] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:05.821022 5654 consensus_peers.cc:485] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.820956 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.821439 5654 leader_election.cc:341] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.822539 5839 tablet_bootstrap.cc:492] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.822741 5839 ts_tablet_manager.cc:1367] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20240507 08:12:05.825078 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 05ed335aadfe432cac42539e196f095a (DEFAULT_TABLE table=default.Table_f5289d2de1e64222b8a1faf354cf0048 [id=24c138ae9237417280d0904ee7ecab00]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.825598 6684 raft_consensus.cc:482] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.825716 6684 raft_consensus.cc:504] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.825927 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 73f82413951c48a89e7db8fe9e50c749 (DEFAULT_TABLE table=default.Table_816c7491958e48f29ae98e8b458aa440 [id=639576b94d4c40419c1b969d30944be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.823529 5839 raft_consensus.cc:348] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.826139 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_75272258956b4c93ac0a4aec66343afc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:05.826287 6684 leader_election.cc:288] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.826287 5839 raft_consensus.cc:374] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.826366 5839 raft_consensus.cc:729] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.826428 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 73f82413951c48a89e7db8fe9e50c749 (DEFAULT_TABLE table=default.Table_816c7491958e48f29ae98e8b458aa440 [id=639576b94d4c40419c1b969d30944be3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.826706 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 73f82413951c48a89e7db8fe9e50c749 (DEFAULT_TABLE table=default.Table_816c7491958e48f29ae98e8b458aa440 [id=639576b94d4c40419c1b969d30944be3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.826709 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75272258956b4c93ac0a4aec66343afc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.827474 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.827728 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 5e05cf76c9934d4881c3296fd6f8328c (DEFAULT_TABLE table=default.Table_73e379813c044602aa406c9ac47557ce [id=44e15ebbbbc540239498672c619348b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.827873 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 5e05cf76c9934d4881c3296fd6f8328c (DEFAULT_TABLE table=default.Table_73e379813c044602aa406c9ac47557ce [id=44e15ebbbbc540239498672c619348b5]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.828006 5377 leader_election.cc:341] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.828243 5839 consensus_queue.cc:260] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.828599 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.828707 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 5e05cf76c9934d4881c3296fd6f8328c (DEFAULT_TABLE table=default.Table_73e379813c044602aa406c9ac47557ce [id=44e15ebbbbc540239498672c619348b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.829723 5443 raft_consensus.cc:2454] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:05.829658 6508 raft_consensus.cc:504] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.830036 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0 (DEFAULT_TABLE table=default.Table_dec376d239c14388b1cdd9fa9b67873f [id=0ebf6df4b68741d6a17792fa0318e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.830247 6508 leader_election.cc:288] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:05.830418 5378 leader_election.cc:341] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.830503 5378 leader_election.cc:302] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.829725 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0 (DEFAULT_TABLE table=default.Table_dec376d239c14388b1cdd9fa9b67873f [id=0ebf6df4b68741d6a17792fa0318e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.830952 6684 raft_consensus.cc:2726] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.831363 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0 (DEFAULT_TABLE table=default.Table_dec376d239c14388b1cdd9fa9b67873f [id=0ebf6df4b68741d6a17792fa0318e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.831465 5656 leader_election.cc:302] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.831702 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.832089 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.832235 5443 raft_consensus.cc:3037] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.834079 6553 raft_consensus.cc:504] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:05.834626 6553 leader_election.cc:288] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:05.834636 5443 raft_consensus.cc:2454] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:05.834761 6553 raft_consensus.cc:2781] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.835065 6553 raft_consensus.cc:686] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:05.835297 6553 consensus_queue.cc:237] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.836392 5656 leader_election.cc:302] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:05.836611 5559 tablet_service.cc:1467] Processing CreateTablet for tablet a6c50d46db8949ee978b87f83a640c0a (DEFAULT_TABLE table=default.Table_c91f44a911c74b99ae52369cf8d517a3 [id=a071b20cbd5d458cbdb8443c4c0e8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.836915 6675 raft_consensus.cc:2781] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:05.837014 6675 raft_consensus.cc:686] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:05.837196 6675 consensus_queue.cc:237] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.837741 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f82413951c48a89e7db8fe9e50c749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.838243 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e05cf76c9934d4881c3296fd6f8328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.838260 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:05.838445 5443 raft_consensus.cc:3037] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.840143 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab43b817cb448259b9c939ede3bcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.840320 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e05cf76c9934d4881c3296fd6f8328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.840509 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ed335aadfe432cac42539e196f095a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.842693 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:05.843274 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e05cf76c9934d4881c3296fd6f8328c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.843713 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.845600 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab43b817cb448259b9c939ede3bcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.846673 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ed335aadfe432cac42539e196f095a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.847141 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f82413951c48a89e7db8fe9e50c749. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.847365 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73f82413951c48a89e7db8fe9e50c749. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:05.851943 5654 leader_election.cc:341] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.856470 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9a8c1adf5d32474c9d849f39a54d8e84" } new_table_name: "default.Table_f0de4c39e4294c98aeb782a829a89f68" modify_external_catalogs: true
I20240507 08:12:05.857739 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4716c1285e91432a92b25ab13ada2779" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:05.857859 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d4098fc03a7c4f77bf81551f3b649102"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.858522 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4098fc03a7c4f77bf81551f3b649102 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.859181 5840 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.859315 5840 ts_tablet_manager.cc:1367] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.001s
W20240507 08:12:05.860864 5654 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.861362 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet c33710c6b55744c4ba0843c228abded3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.862432 5840 raft_consensus.cc:348] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.862653 5840 raft_consensus.cc:374] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.862715 5840 raft_consensus.cc:729] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.865552 5840 consensus_queue.cc:260] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.866957 5840 ts_tablet_manager.cc:1398] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240507 08:12:05.867213 5840 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.869120 5839 ts_tablet_manager.cc:1398] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240507 08:12:05.874100 5840 tablet_bootstrap.cc:654] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.874791 5839 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:05.879631 5839 tablet_bootstrap.cc:654] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.880867 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ab43b817cb448259b9c939ede3bcb6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.881079 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c56bd481b4d4c8285e70d9b01f9e8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.881537 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3618b416194f4effb29b823709593359
I20240507 08:12:05.881690 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05ed335aadfe432cac42539e196f095a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.882191 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 3618b416194f4effb29b823709593359 deleted and replaced by tablet fda4b1ceaf0b4f3385258f3703e4461b
I20240507 08:12:05.883481 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c50d46db8949ee978b87f83a640c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.884025 5411 tablet_service.cc:1514] Processing DeleteTablet for tablet 3618b416194f4effb29b823709593359 with delete_type TABLET_DATA_DELETED (replaced by tablet fda4b1ceaf0b4f3385258f3703e4461b) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:05.887547 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 3618b416194f4effb29b823709593359 with delete_type TABLET_DATA_DELETED (replaced by tablet fda4b1ceaf0b4f3385258f3703e4461b) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:05.887836 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3618b416194f4effb29b823709593359 with delete_type TABLET_DATA_DELETED (replaced by tablet fda4b1ceaf0b4f3385258f3703e4461b) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.890604 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a69f8756b5934280bf92b31a383e0f88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.891281 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a69f8756b5934280bf92b31a383e0f88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.892846 5695 tablet_service.cc:1467] Processing CreateTablet for tablet a6c50d46db8949ee978b87f83a640c0a (DEFAULT_TABLE table=default.Table_c91f44a911c74b99ae52369cf8d517a3 [id=a071b20cbd5d458cbdb8443c4c0e8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.894366 5411 tablet_service.cc:1467] Processing CreateTablet for tablet a6c50d46db8949ee978b87f83a640c0a (DEFAULT_TABLE table=default.Table_c91f44a911c74b99ae52369cf8d517a3 [id=a071b20cbd5d458cbdb8443c4c0e8a36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.896617 5416 tablet_service.cc:1467] Processing CreateTablet for tablet dfd9624f4337437bb11b247048e28c08 (DEFAULT_TABLE table=default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed [id=81b147125efb414a85615162da7fa7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.896901 6508 raft_consensus.cc:482] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.896900 5551 tablet_service.cc:1467] Processing CreateTablet for tablet dfd9624f4337437bb11b247048e28c08 (DEFAULT_TABLE table=default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed [id=81b147125efb414a85615162da7fa7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.897027 6508 raft_consensus.cc:504] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.897817 6508 leader_election.cc:288] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.902487 5688 tablet_service.cc:1467] Processing CreateTablet for tablet dfd9624f4337437bb11b247048e28c08 (DEFAULT_TABLE table=default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed [id=81b147125efb414a85615162da7fa7b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.902720 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a425a90f0449bcaaacb4c3f653d26c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.903440 5843 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.903579 5843 ts_tablet_manager.cc:1367] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.003s
W20240507 08:12:05.903649 5654 leader_election.cc:341] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.903961 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.904174 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.904305 5684 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.904310 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:05.905229 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a425a90f0449bcaaacb4c3f653d26c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.905405 5444 raft_consensus.cc:2454] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.906016 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_83aa0d0f56b448378d9c58f5d34035fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:05.906539 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83aa0d0f56b448378d9c58f5d34035fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:05.907594 5656 leader_election.cc:302] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.907824 6553 raft_consensus.cc:2781] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.907902 6553 raft_consensus.cc:482] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.907961 6553 raft_consensus.cc:3037] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.909783 5684 tablet_service.cc:1467] Processing CreateTablet for tablet dae98482d16942a7b630462d9147855e (DEFAULT_TABLE table=default.Table_14b7abf57dc043ceb834e7e62ff05288 [id=7fe3e5b0569b44d6aea493b891e73f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.911427 5425 tablet_service.cc:1467] Processing CreateTablet for tablet dae98482d16942a7b630462d9147855e (DEFAULT_TABLE table=default.Table_14b7abf57dc043ceb834e7e62ff05288 [id=7fe3e5b0569b44d6aea493b891e73f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.912097 5843 raft_consensus.cc:348] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.912269 5843 raft_consensus.cc:374] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.912309 5843 raft_consensus.cc:729] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.912863 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 (DEFAULT_TABLE table=default.Table_452e7072fe42490f92423745f0bf9d07 [id=68a80f7012044e56a186534cead38f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.912504 5843 consensus_queue.cc:260] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.913621 5843 ts_tablet_manager.cc:1398] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:12:05.914853 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 (DEFAULT_TABLE table=default.Table_452e7072fe42490f92423745f0bf9d07 [id=68a80f7012044e56a186534cead38f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.915809 5843 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.916399 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae2c63d7d540148c352f478e8d9bac (DEFAULT_TABLE table=default.Table_afca694c460546b890018320967f9928 [id=d751b1b357ca416e952d754213d906c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.917623 6743 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:05.918242 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae2c63d7d540148c352f478e8d9bac (DEFAULT_TABLE table=default.Table_afca694c460546b890018320967f9928 [id=d751b1b357ca416e952d754213d906c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.918591 6603 raft_consensus.cc:482] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.918738 6603 raft_consensus.cc:504] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.919173 5843 tablet_bootstrap.cc:654] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.919368 6603 leader_election.cc:288] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.920094 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 0f42c1cee62b478eb653022b8c9b657d (DEFAULT_TABLE table=default.Table_d692c7e6177c4bf487d2f91bbe15a9d9 [id=5529cf9e5cd740898d814b20e4cdfa05]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.921521 5520 leader_election.cc:341] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: c33710c6b55744c4ba0843c228abded3
I20240507 08:12:05.921967 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 0f42c1cee62b478eb653022b8c9b657d (DEFAULT_TABLE table=default.Table_d692c7e6177c4bf487d2f91bbe15a9d9 [id=5529cf9e5cd740898d814b20e4cdfa05]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:05.922756 5517 leader_election.cc:341] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: c33710c6b55744c4ba0843c228abded3
I20240507 08:12:05.922850 5517 leader_election.cc:302] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:05.923236 6603 raft_consensus.cc:2726] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.923697 5682 tablet_service.cc:1467] Processing CreateTablet for tablet bf0dc818b10a4e1e921452a83a5c05ee (DEFAULT_TABLE table=default.Table_c373339c2d25449e8c18ce24b35e434e [id=06a0ce1fcb1e4caf8f8d2bdc8f584cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.926201 5427 tablet_service.cc:1467] Processing CreateTablet for tablet bf0dc818b10a4e1e921452a83a5c05ee (DEFAULT_TABLE table=default.Table_c373339c2d25449e8c18ce24b35e434e [id=06a0ce1fcb1e4caf8f8d2bdc8f584cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.927465 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 574a47894d274360891253534bf215a5 (DEFAULT_TABLE table=default.Table_f1b5b7cea20745ecb2e767e67c4c8b38 [id=905a9871b9ff491ebfd888477aebbc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.929081 5840 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.929276 5840 ts_tablet_manager.cc:1367] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.004s
I20240507 08:12:05.929344 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 574a47894d274360891253534bf215a5 (DEFAULT_TABLE table=default.Table_f1b5b7cea20745ecb2e767e67c4c8b38 [id=905a9871b9ff491ebfd888477aebbc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.930213 5840 raft_consensus.cc:348] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.930655 5840 raft_consensus.cc:374] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.930759 5840 raft_consensus.cc:729] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.931034 5840 consensus_queue.cc:260] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.932596 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 (DEFAULT_TABLE table=default.Table_35569f4031ae4c15a8f2653100e5551b [id=ecb500f78df24a71816101c0bd9c6a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.933806 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 (DEFAULT_TABLE table=default.Table_35569f4031ae4c15a8f2653100e5551b [id=ecb500f78df24a71816101c0bd9c6a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.934104 6614 raft_consensus.cc:504] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.934502 5692 tablet_service.cc:1467] Processing CreateTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c (DEFAULT_TABLE table=default.Table_8dd482935ec64031b96ef89534429ac3 [id=4fcf3a25a3224375a5599f345405a679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.934633 5840 ts_tablet_manager.cc:1398] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240507 08:12:05.934738 6614 leader_election.cc:288] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.934893 5840 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.935910 5409 tablet_service.cc:1467] Processing CreateTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c (DEFAULT_TABLE table=default.Table_8dd482935ec64031b96ef89534429ac3 [id=4fcf3a25a3224375a5599f345405a679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.936232 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe721ba2f0ed4a3a887e7496e85d163c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.936765 5840 tablet_bootstrap.cc:654] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.938911 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f42c1cee62b478eb653022b8c9b657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.939291 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0c793dbc884efd92b49ae7798206a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.940454 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae98482d16942a7b630462d9147855e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.940863 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81fcf82e60c94e9eaadd75bd37595261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.941356 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a47894d274360891253534bf215a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.941442 6547 ts_tablet_manager.cc:1888] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:05.941756 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.942098 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae2c63d7d540148c352f478e8d9bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.942456 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd9624f4337437bb11b247048e28c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.942813 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c50d46db8949ee978b87f83a640c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.943177 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.943326 5840 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.943434 5840 ts_tablet_manager.cc:1367] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.002s
I20240507 08:12:05.944512 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 (DEFAULT_TABLE table=default.Table_e57fcc9d421a4abdb75cb707e06febbe [id=11283fbad3b946d8a2252a4fcf2bafab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.944826 5840 raft_consensus.cc:348] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.945017 5840 raft_consensus.cc:374] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.945075 5840 raft_consensus.cc:729] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.945313 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 (DEFAULT_TABLE table=default.Table_e57fcc9d421a4abdb75cb707e06febbe [id=11283fbad3b946d8a2252a4fcf2bafab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.946344 5840 consensus_queue.cc:260] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.946839 5840 ts_tablet_manager.cc:1398] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:05.947085 5840 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.947084 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a85df85b49db4145ae58e672786dca62 (DEFAULT_TABLE table=default.Table_e72c7ad89d294c4caba3c321f723cf2a [id=2ae7723631594107981c7b07a7e3871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.948701 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 90dcd7eee62b458ea4661a05a1bfcac4 (DEFAULT_TABLE table=default.Table_510d804e03f54c96b5321a44a0b1dd38 [id=d89292afebd24ce5a7f40f22d657f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.949815 5681 tablet_service.cc:1467] Processing CreateTablet for tablet a85df85b49db4145ae58e672786dca62 (DEFAULT_TABLE table=default.Table_e72c7ad89d294c4caba3c321f723cf2a [id=2ae7723631594107981c7b07a7e3871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.949882 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 90dcd7eee62b458ea4661a05a1bfcac4 (DEFAULT_TABLE table=default.Table_510d804e03f54c96b5321a44a0b1dd38 [id=d89292afebd24ce5a7f40f22d657f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.950884 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1df8f1ec4347f49746d1c48c767b58 (DEFAULT_TABLE table=default.Table_50c935f08e4f4f4caeb2643cf499aace [id=64d764a4dd9f41ed88914a479fa62ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.951494 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1df8f1ec4347f49746d1c48c767b58 (DEFAULT_TABLE table=default.Table_50c935f08e4f4f4caeb2643cf499aace [id=64d764a4dd9f41ed88914a479fa62ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.952582 5687 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a7c1bee444c75a4ed6215db0e25bd (DEFAULT_TABLE table=default.Table_02c584c9b6774d37b485b35ba37050ec [id=6f4324b544004a9f834afc787640bc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.953229 5420 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a7c1bee444c75a4ed6215db0e25bd (DEFAULT_TABLE table=default.Table_02c584c9b6774d37b485b35ba37050ec [id=6f4324b544004a9f834afc787640bc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.954550 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 389aced831b547f6af5c3eff90c4d3b2 (DEFAULT_TABLE table=default.Table_cc196bc1f789402ab1d5fbb907534f09 [id=96a94646a959426687498a5ed08de322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.955193 5840 tablet_bootstrap.cc:654] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.956655 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc71de9f20547c8a62d320b11b4c0f9 (DEFAULT_TABLE table=default.Table_78c7e392d6e644cea15ddcdcb2e991aa [id=f18192375d1d4c40a6d0dd80faacf407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.956784 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c978ec8bca4aa5b2ffaf22c3290747" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:05.956950 5720 raft_consensus.cc:3037] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.958488 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac55233449844d2bd40fee0be427e13 (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:05.961084 5840 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:05.961208 5840 ts_tablet_manager.cc:1367] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.001s
I20240507 08:12:05.941546 6547 log.cc:1196] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/fde460fdcf254af6a3cdaaac22ac2d93
I20240507 08:12:05.961948 6547 ts_tablet_manager.cc:1909] T fde460fdcf254af6a3cdaaac22ac2d93 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:05.962360 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0dc818b10a4e1e921452a83a5c05ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:05.968657 5840 raft_consensus.cc:348] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.968839 5840 raft_consensus.cc:374] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.968885 5840 raft_consensus.cc:729] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.969067 5840 consensus_queue.cc:260] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.970307 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97c978ec8bca4aa5b2ffaf22c3290747" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:05.970414 6684 raft_consensus.cc:482] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.970522 6684 raft_consensus.cc:504] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.971036 6684 leader_election.cc:288] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.973999 6614 raft_consensus.cc:482] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.974129 6614 raft_consensus.cc:504] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.974640 6614 leader_election.cc:288] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.969543 5840 ts_tablet_manager.cc:1398] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240507 08:12:05.971776 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.973834 6508 raft_consensus.cc:482] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.978552 6508 raft_consensus.cc:504] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.979081 6508 leader_election.cc:288] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.976055 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:05.973858 6675 raft_consensus.cc:482] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:05.980230 5843 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:05.980165 6675 raft_consensus.cc:504] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.980343 5843 ts_tablet_manager.cc:1367] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20240507 08:12:05.980698 6675 leader_election.cc:288] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.981272 5843 raft_consensus.cc:348] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.981453 5843 raft_consensus.cc:374] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.981494 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027b4cefa4d41659f819ee4326470c4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.981699 5377 leader_election.cc:341] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.981649 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.981881 5377 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.981997 5843 raft_consensus.cc:729] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.982072 6684 raft_consensus.cc:482] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:05.982065 5654 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:05.982226 5654 leader_election.cc:341] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.982167 6684 raft_consensus.cc:504] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.982209 5843 consensus_queue.cc:260] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.981329 5840 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:05.982671 6684 leader_election.cc:288] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:05.982849 5843 ts_tablet_manager.cc:1398] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240507 08:12:05.983261 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:05.983703 5377 leader_election.cc:341] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.983109 5843 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:05.987190 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:05.987428 5447 raft_consensus.cc:2454] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:05.987701 5843 tablet_bootstrap.cc:654] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:05.985383 5840 tablet_bootstrap.cc:654] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:05.987895 5656 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7027b4cefa4d41659f819ee4326470c4
I20240507 08:12:05.988020 5656 leader_election.cc:302] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:05.988337 5378 leader_election.cc:341] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.988417 5656 leader_election.cc:302] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.988837 6675 raft_consensus.cc:2726] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.988866 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.988996 6675 raft_consensus.cc:2781] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:05.989073 6675 raft_consensus.cc:482] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:05.989136 6675 raft_consensus.cc:3037] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:05.989193 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:05.990058 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:05.990989 5378 leader_election.cc:341] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.991086 5378 leader_election.cc:302] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:05.991284 5654 consensus_peers.cc:485] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:05.991302 6614 raft_consensus.cc:2726] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:05.991760 5378 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.991847 5378 leader_election.cc:302] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.992017 6614 raft_consensus.cc:2726] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:05.993011 5378 leader_election.cc:341] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:05.993098 5378 leader_election.cc:302] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:05.994748 6614 raft_consensus.cc:2726] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:05.996188 5839 tablet_bootstrap.cc:492] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:05.996305 5839 ts_tablet_manager.cc:1367] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.009s
I20240507 08:12:05.996964 5839 raft_consensus.cc:348] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.997133 5839 raft_consensus.cc:374] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:05.997189 5839 raft_consensus.cc:729] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:05.997362 5839 consensus_queue.cc:260] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.998302 6553 raft_consensus.cc:504] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:05.998895 6553 leader_election.cc:288] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:05.999264 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a425a90f0449bcaaacb4c3f653d26c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:05.999662 5654 leader_election.cc:341] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.000330 5840 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.000439 5840 ts_tablet_manager.cc:1367] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240507 08:12:06.001055 5548 tablet_service.cc:1467] Processing CreateTablet for tablet dae98482d16942a7b630462d9147855e (DEFAULT_TABLE table=default.Table_14b7abf57dc043ceb834e7e62ff05288 [id=7fe3e5b0569b44d6aea493b891e73f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.001077 5840 raft_consensus.cc:348] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.001231 5840 raft_consensus.cc:374] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.001286 5840 raft_consensus.cc:729] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.003125 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a425a90f0449bcaaacb4c3f653d26c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.003284 5447 raft_consensus.cc:3037] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.005023 5549 tablet_service.cc:1467] Processing CreateTablet for tablet bf0dc818b10a4e1e921452a83a5c05ee (DEFAULT_TABLE table=default.Table_c373339c2d25449e8c18ce24b35e434e [id=06a0ce1fcb1e4caf8f8d2bdc8f584cef]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.005909 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 (DEFAULT_TABLE table=default.Table_35569f4031ae4c15a8f2653100e5551b [id=ecb500f78df24a71816101c0bd9c6a2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.006287 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 574a47894d274360891253534bf215a5 (DEFAULT_TABLE table=default.Table_f1b5b7cea20745ecb2e767e67c4c8b38 [id=905a9871b9ff491ebfd888477aebbc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.006568 5550 tablet_service.cc:1467] Processing CreateTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c (DEFAULT_TABLE table=default.Table_8dd482935ec64031b96ef89534429ac3 [id=4fcf3a25a3224375a5599f345405a679]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.006753 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.006845 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 12e85df8b900468ba2584fd8f2e9bb2d
I20240507 08:12:06.007180 5840 consensus_queue.cc:260] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.007782 5546 tablet_service.cc:1467] Processing CreateTablet for tablet a85df85b49db4145ae58e672786dca62 (DEFAULT_TABLE table=default.Table_e72c7ad89d294c4caba3c321f723cf2a [id=2ae7723631594107981c7b07a7e3871e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.009563 5840 ts_tablet_manager.cc:1398] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20240507 08:12:06.010545 5843 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.010685 5843 ts_tablet_manager.cc:1367] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.001s
I20240507 08:12:06.011353 5843 raft_consensus.cc:348] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.011533 5843 raft_consensus.cc:374] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.011584 5843 raft_consensus.cc:729] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.011782 5843 consensus_queue.cc:260] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.012266 5843 ts_tablet_manager.cc:1398] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:06.012472 5843 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.013989 6553 raft_consensus.cc:482] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.014040 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 0f42c1cee62b478eb653022b8c9b657d (DEFAULT_TABLE table=default.Table_d692c7e6177c4bf487d2f91bbe15a9d9 [id=5529cf9e5cd740898d814b20e4cdfa05]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.014261 6553 raft_consensus.cc:504] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.014559 5843 tablet_bootstrap.cc:654] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.015506 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.015988 5654 leader_election.cc:341] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.016165 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 (DEFAULT_TABLE table=default.Table_452e7072fe42490f92423745f0bf9d07 [id=68a80f7012044e56a186534cead38f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.015427 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 11ae2c63d7d540148c352f478e8d9bac (DEFAULT_TABLE table=default.Table_afca694c460546b890018320967f9928 [id=d751b1b357ca416e952d754213d906c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.016906 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.017081 5446 raft_consensus.cc:2454] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.017472 5656 leader_election.cc:302] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.017726 6508 raft_consensus.cc:2781] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.017793 6508 raft_consensus.cc:482] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.017850 6508 raft_consensus.cc:3037] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.018501 6553 leader_election.cc:288] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.020396 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 9a1df8f1ec4347f49746d1c48c767b58 (DEFAULT_TABLE table=default.Table_50c935f08e4f4f4caeb2643cf499aace [id=64d764a4dd9f41ed88914a479fa62ad3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.021456 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 90dcd7eee62b458ea4661a05a1bfcac4 (DEFAULT_TABLE table=default.Table_510d804e03f54c96b5321a44a0b1dd38 [id=d89292afebd24ce5a7f40f22d657f6f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.015491 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 (DEFAULT_TABLE table=default.Table_e57fcc9d421a4abdb75cb707e06febbe [id=11283fbad3b946d8a2252a4fcf2bafab]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.021922 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd9624f4337437bb11b247048e28c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.022109 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85df85b49db4145ae58e672786dca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.022262 6763 consensus_queue.cc:1035] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:06.022724 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1df8f1ec4347f49746d1c48c767b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.023088 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dcd7eee62b458ea4661a05a1bfcac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.025665 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0dc818b10a4e1e921452a83a5c05ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.025863 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe721ba2f0ed4a3a887e7496e85d163c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.026252 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a47894d274360891253534bf215a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.029125 5443 raft_consensus.cc:2454] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.029196 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01820ecad55a4097bc5917a8f7949fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.030534 5656 leader_election.cc:302] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.030959 6763 raft_consensus.cc:2781] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.034312 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 717) took 1194 ms. Trace:
I20240507 08:12:06.034482 5843 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.034394 6547 rpcz_store.cc:276] 0507 08:12:04.839908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.842749 (+ 2841us) service_pool.cc:225] Handling call
0507 08:12:05.746125 (+903376us) ts_tablet_manager.cc:1120] Deleting tablet fde460fdcf254af6a3cdaaac22ac2d93
0507 08:12:05.746131 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.837648 (+ 91517us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:05.941319 (+103671us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.034292 (+ 92973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":903205}
I20240507 08:12:06.034591 5843 ts_tablet_manager.cc:1367] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.000s
I20240507 08:12:06.034689 6547 tablet_replica.cc:330] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:06.034865 6547 raft_consensus.cc:2229] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.035329 5843 raft_consensus.cc:348] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.035496 5843 raft_consensus.cc:374] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.035550 5843 raft_consensus.cc:729] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.035838 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81fcf82e60c94e9eaadd75bd37595261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.035998 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389aced831b547f6af5c3eff90c4d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.036499 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae2c63d7d540148c352f478e8d9bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.036664 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 574a47894d274360891253534bf215a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.036942 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0c793dbc884efd92b49ae7798206a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.037282 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f42c1cee62b478eb653022b8c9b657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.037635 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01820ecad55a4097bc5917a8f7949fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.037855 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac55233449844d2bd40fee0be427e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.038192 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85df85b49db4145ae58e672786dca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.038491 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dcd7eee62b458ea4661a05a1bfcac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.038738 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7a7c1bee444c75a4ed6215db0e25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.039122 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6c50d46db8949ee978b87f83a640c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.039278 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe721ba2f0ed4a3a887e7496e85d163c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.039322 5720 raft_consensus.cc:2454] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:06.039454 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfd9624f4337437bb11b247048e28c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.037955 5843 consensus_queue.cc:260] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.039795 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae98482d16942a7b630462d9147855e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.040035 5843 ts_tablet_manager.cc:1398] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:06.040164 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0dc818b10a4e1e921452a83a5c05ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.040365 5843 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.040520 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6f752a0d4485e9606a5570a0d6c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.040881 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1df8f1ec4347f49746d1c48c767b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.041277 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc71de9f20547c8a62d320b11b4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.046694 5377 leader_election.cc:302] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.047184 6753 raft_consensus.cc:2781] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.047299 6753 raft_consensus.cc:686] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:06.047505 6753 consensus_queue.cc:237] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.048576 5839 ts_tablet_manager.cc:1398] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.052s user 0.000s sys 0.001s
I20240507 08:12:06.049031 5839 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.050616 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b0c793dbc884efd92b49ae7798206a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.050817 5839 tablet_bootstrap.cc:654] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.052803 5843 tablet_bootstrap.cc:654] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.054167 6547 raft_consensus.cc:2258] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.054659 6547 ts_tablet_manager.cc:1875] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.059387 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6f752a0d4485e9606a5570a0d6c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.059603 6553 raft_consensus.cc:482] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.059728 6553 raft_consensus.cc:504] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.060715 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.061281 5654 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.062331 5656 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e555a89d340448bc852f2441ca850213
I20240507 08:12:06.062443 5656 leader_election.cc:302] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.062623 6553 leader_election.cc:288] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.062669 5841 ts_tablet_manager.cc:1888] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.062793 5841 log.cc:1196] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4fcc1d7e309946d783a524b61d5f1269
I20240507 08:12:06.062841 6553 raft_consensus.cc:2726] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.063153 5841 ts_tablet_manager.cc:1909] T 4fcc1d7e309946d783a524b61d5f1269 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:06.066928 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01820ecad55a4097bc5917a8f7949fe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.067433 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a85df85b49db4145ae58e672786dca62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.067821 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a1df8f1ec4347f49746d1c48c767b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.068868 6553 raft_consensus.cc:482] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.069020 6553 raft_consensus.cc:504] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.070175 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.070679 5654 leader_election.cc:341] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.071566 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.071751 5446 raft_consensus.cc:2454] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.072171 5656 leader_election.cc:302] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.072438 6553 leader_election.cc:288] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.072464 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b28dbe67129643cfad8ac14f10aa72c5" } new_table_name: "default.Table_b0948279632f4c32a83404acd89ae917" modify_external_catalogs: true
I20240507 08:12:06.072693 6553 raft_consensus.cc:482] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.072800 6553 raft_consensus.cc:504] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.073071 6734 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:06.073555 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.073885 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c7d19b7cc39a4337826819a20205aecc
W20240507 08:12:06.074028 5654 leader_election.cc:341] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.074560 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_94c6194476954481ab568218481b03c8" } modify_external_catalogs: true
I20240507 08:12:06.074755 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_94c6194476954481ab568218481b03c8" } modify_external_catalogs: true
I20240507 08:12:06.074816 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.074990 5444 raft_consensus.cc:2454] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.075332 5656 leader_election.cc:302] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.076507 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d228b6629c7349a5843e85ab6a55e3bd" } modify_external_catalogs: true
I20240507 08:12:06.076735 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d228b6629c7349a5843e85ab6a55e3bd" } modify_external_catalogs: true
W20240507 08:12:06.078284 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 consensus state has no leader
I20240507 08:12:06.078707 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_83eabeb1c35e407cb08ed3a030480d7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.079340 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83eabeb1c35e407cb08ed3a030480d7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.080825 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet f513573e340547d6ae5ab848d00ace72 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:06.082082 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cd8f372c67b74794a5ac56adc1990bad" } new_table_name: "default.Table_ad611c4e3e3543fd98e93440daea8098" modify_external_catalogs: true
I20240507 08:12:06.083674 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_36a39b6cb68c403d8d67600517f6bb5b" } new_table_name: "default.Table_b7a2117351ac420884d522481779f3fd" modify_external_catalogs: true
I20240507 08:12:06.085032 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17c6f752a0d4485e9606a5570a0d6c96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.085590 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90dcd7eee62b458ea4661a05a1bfcac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.085609 6764 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:06.086033 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7a7c1bee444c75a4ed6215db0e25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.086217 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc71de9f20547c8a62d320b11b4c0f9 (DEFAULT_TABLE table=default.Table_78c7e392d6e644cea15ddcdcb2e991aa [id=f18192375d1d4c40a6d0dd80faacf407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.086477 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac55233449844d2bd40fee0be427e13 (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.086994 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce7a166001486197b78f8b02a437ed (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.087536 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 389aced831b547f6af5c3eff90c4d3b2 (DEFAULT_TABLE table=default.Table_cc196bc1f789402ab1d5fbb907534f09 [id=96a94646a959426687498a5ed08de322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.088032 5546 tablet_service.cc:1467] Processing CreateTablet for tablet fe7a7c1bee444c75a4ed6215db0e25bd (DEFAULT_TABLE table=default.Table_02c584c9b6774d37b485b35ba37050ec [id=6f4324b544004a9f834afc787640bc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.094395 6553 leader_election.cc:288] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.094564 6553 raft_consensus.cc:2781] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.094621 6553 raft_consensus.cc:482] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.094671 6553 raft_consensus.cc:3037] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.095532 6603 raft_consensus.cc:482] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.095652 6603 raft_consensus.cc:504] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.096221 6603 leader_election.cc:288] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.096752 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.097548 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3d3baa99b74704ab68278b1437d5fb (DEFAULT_TABLE table=default.Table_9ab22bf322414792b4044b2f77b6144a [id=ebcdc2f8fffd4bbe9fbbe9193e9f8937]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.104583 6753 raft_consensus.cc:482] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.104743 6753 raft_consensus.cc:504] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.104825 6779 raft_consensus.cc:2781] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.104918 6779 raft_consensus.cc:482] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.105003 6779 raft_consensus.cc:3037] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.105932 6753 leader_election.cc:288] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.107514 5839 tablet_bootstrap.cc:492] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.107658 5839 ts_tablet_manager.cc:1367] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.059s user 0.007s sys 0.000s
I20240507 08:12:06.107668 5840 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:06.108322 5839 raft_consensus.cc:348] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.108500 5839 raft_consensus.cc:374] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.108547 5839 raft_consensus.cc:729] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.108755 5839 consensus_queue.cc:260] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.109195 5839 ts_tablet_manager.cc:1398] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240507 08:12:06.110589 6763 raft_consensus.cc:686] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.110687 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc74342c8aee4017a212aa88a7f3dc06" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.116067 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0ce315fc8bc94622b827617e4cd26c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:06.116111 6763 consensus_queue.cc:237] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:06.116611 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ce315fc8bc94622b827617e4cd26c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.117918 5517 leader_election.cc:302] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.118170 6603 raft_consensus.cc:2726] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.118666 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.118813 5840 tablet_bootstrap.cc:654] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:06.120241 5377 leader_election.cc:341] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.120298 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:06.120756 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9ced879f35e14106b2609c78b7f24002
I20240507 08:12:06.121495 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_70cacbd76a194d649516d56327e82ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.122031 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_70cacbd76a194d649516d56327e82ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.123485 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_ee5279fc19a046aebae1f622389f59a2 [id=c6a13f9902b0424ca8df2184779a407c] alter complete (version 3)
I20240507 08:12:06.124045 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a16c963f4cf149fd9ff050aab7c68dbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.124498 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a16c963f4cf149fd9ff050aab7c68dbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.127645 6753 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.127774 6753 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.128334 6753 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.132292 6603 raft_consensus.cc:482] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.132421 6603 raft_consensus.cc:504] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.132673 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.132081 5557 tablet_service.cc:1467] Processing CreateTablet for tablet c6a25b41cf0d45f9bc3b2d9731a10da3 (DEFAULT_TABLE table=default.Table_493ded421178438798c30ff021b4531b [id=0a1a2bcfcb4448a4a963e143b01fc906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.133011 6603 leader_election.cc:288] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.133411 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.133862 5520 leader_election.cc:341] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5554ebc621c24d459668f2cefc20b641
I20240507 08:12:06.134461 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.137475 5447 raft_consensus.cc:2454] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.138288 5656 leader_election.cc:302] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.138703 6773 raft_consensus.cc:2781] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.138832 6773 raft_consensus.cc:686] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.139156 6773 consensus_queue.cc:237] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.140304 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 389aced831b547f6af5c3eff90c4d3b2 (DEFAULT_TABLE table=default.Table_cc196bc1f789402ab1d5fbb907534f09 [id=96a94646a959426687498a5ed08de322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.140671 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389aced831b547f6af5c3eff90c4d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.143857 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 4dc71de9f20547c8a62d320b11b4c0f9 (DEFAULT_TABLE table=default.Table_78c7e392d6e644cea15ddcdcb2e991aa [id=f18192375d1d4c40a6d0dd80faacf407]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.144250 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc71de9f20547c8a62d320b11b4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.153484 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce7a166001486197b78f8b02a437ed (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.153944 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce7a166001486197b78f8b02a437ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.157794 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c6a25b41cf0d45f9bc3b2d9731a10da3 (DEFAULT_TABLE table=default.Table_493ded421178438798c30ff021b4531b [id=0a1a2bcfcb4448a4a963e143b01fc906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.158356 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a25b41cf0d45f9bc3b2d9731a10da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.160629 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3d3baa99b74704ab68278b1437d5fb (DEFAULT_TABLE table=default.Table_9ab22bf322414792b4044b2f77b6144a [id=ebcdc2f8fffd4bbe9fbbe9193e9f8937]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.160941 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3d3baa99b74704ab68278b1437d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.162197 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.162382 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d19b7cc39a4337826819a20205aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.162525 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ced879f35e14106b2609c78b7f24002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.164583 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 4ac55233449844d2bd40fee0be427e13 (DEFAULT_TABLE table=default.Table_67e5969a8d434e50bcd09e39a824f5c8 [id=3273c97f16d6408f9c12e4a04711be33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.164932 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac55233449844d2bd40fee0be427e13. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:06.165201 5377 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.169080 5839 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.170511 5839 tablet_bootstrap.cc:654] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:06.179903 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 119074us
I20240507 08:12:06.190182 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dae98482d16942a7b630462d9147855e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.190398 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4dc71de9f20547c8a62d320b11b4c0f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.190593 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ac55233449844d2bd40fee0be427e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.191217 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce7a166001486197b78f8b02a437ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.191571 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 389aced831b547f6af5c3eff90c4d3b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.191658 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet fde460fdcf254af6a3cdaaac22ac2d93 (table default.Table_30a40b16a51c4ad28c5fee73ed5e4185 [id=d78e372a58684acb9dc35c7a3ff2b2c9]) successfully deleted
I20240507 08:12:06.191946 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe7a7c1bee444c75a4ed6215db0e25bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.192274 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a25b41cf0d45f9bc3b2d9731a10da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.192668 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11ae2c63d7d540148c352f478e8d9bac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.193145 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81fcf82e60c94e9eaadd75bd37595261. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.201720 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f42c1cee62b478eb653022b8c9b657d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.206296 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ced879f35e14106b2609c78b7f24002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:06.206408 5378 leader_election.cc:341] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.206532 5378 leader_election.cc:302] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.206549 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.207262 6684 raft_consensus.cc:2726] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.208014 6675 raft_consensus.cc:504] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.208763 6675 leader_election.cc:288] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.209725 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 725) took 1812 ms. Trace:
I20240507 08:12:06.209806 5841 rpcz_store.cc:276] 0507 08:12:04.396964 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.397592 (+ 628us) service_pool.cc:225] Handling call
0507 08:12:05.736602 (+1339010us) ts_tablet_manager.cc:1120] Deleting tablet 4fcc1d7e309946d783a524b61d5f1269
0507 08:12:05.748825 (+ 12223us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.935007 (+186182us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.062533 (+127526us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.209703 (+147170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1338849}
I20240507 08:12:06.210143 5841 tablet_replica.cc:330] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:06.210306 5841 raft_consensus.cc:2229] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:06.210577 5841 raft_consensus.cc:2258] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.211129 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.211298 5447 raft_consensus.cc:3037] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.212013 5841 ts_tablet_manager.cc:1875] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.213268 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4fcc1d7e309946d783a524b61d5f1269 (table default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]) successfully deleted
W20240507 08:12:06.222069 5378 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.222216 5378 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.222698 6753 raft_consensus.cc:2726] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.223757 6603 raft_consensus.cc:482] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.223894 6603 raft_consensus.cc:504] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.224311 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 38ce7a166001486197b78f8b02a437ed (DEFAULT_TABLE table=default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.224517 6603 leader_election.cc:288] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:06.225517 5520 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:12:06.225826 5447 raft_consensus.cc:2454] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:06.229964 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b340567fcccb48a495033c0c430b33b0 consensus state has no leader
I20240507 08:12:06.229949 6553 raft_consensus.cc:504] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.230537 6553 leader_election.cc:288] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.233193 6779 raft_consensus.cc:504] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.233561 6513 ts_tablet_manager.cc:1888] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.233719 6513 log.cc:1196] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:12:06.234011 6513 ts_tablet_manager.cc:1909] T b22334378eec4ab3885d0f9d2d812a3e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:06.233996 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8d06498c8a2a4d74a95609d3c3c19795"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:06.234295 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3d3baa99b74704ab68278b1437d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:06.234658 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d06498c8a2a4d74a95609d3c3c19795 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.235836 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6def4bf6a7554a4090a39b1823c4097c
I20240507 08:12:06.236176 5697 tablet_service.cc:1467] Processing CreateTablet for tablet c6a25b41cf0d45f9bc3b2d9731a10da3 (DEFAULT_TABLE table=default.Table_493ded421178438798c30ff021b4531b [id=0a1a2bcfcb4448a4a963e143b01fc906]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.236541 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a25b41cf0d45f9bc3b2d9731a10da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.237078 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b81673c2cce64edba3bbf1a45e6c3e49" } new_table_name: "default.Table_323beab207874b98a5785e0dd1f9efcb" modify_external_catalogs: true
I20240507 08:12:06.237967 5656 leader_election.cc:302] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:06.238310 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d33a3ccd7a5e4b9e956735443d4c484a
I20240507 08:12:06.239378 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.239565 5447 raft_consensus.cc:3037] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.241218 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a77bc176ba0948c796f99b08bcc16d1d" } modify_external_catalogs: true
I20240507 08:12:06.241401 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a77bc176ba0948c796f99b08bcc16d1d" } modify_external_catalogs: true
I20240507 08:12:06.242091 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3d3baa99b74704ab68278b1437d5fb (DEFAULT_TABLE table=default.Table_9ab22bf322414792b4044b2f77b6144a [id=ebcdc2f8fffd4bbe9fbbe9193e9f8937]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:06.241716 5517 leader_election.cc:341] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5554ebc621c24d459668f2cefc20b641
I20240507 08:12:06.242314 5517 leader_election.cc:302] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.242421 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3d3baa99b74704ab68278b1437d5fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.242544 6603 raft_consensus.cc:2726] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.243237 5839 tablet_bootstrap.cc:492] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.243350 5839 ts_tablet_manager.cc:1367] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.000s
W20240507 08:12:06.243378 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f64e338784704312b986762a143dda4c consensus state has no leader
I20240507 08:12:06.243834 6784 raft_consensus.cc:2781] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.244351 6780 raft_consensus.cc:482] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.244381 5839 raft_consensus.cc:348] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.244541 5839 raft_consensus.cc:374] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.244598 5839 raft_consensus.cc:729] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:06.244649 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb5a077be5574cb8bc7a2347c6d3af00 has no consensus state
I20240507 08:12:06.244774 5839 consensus_queue.cc:260] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.245451 5839 ts_tablet_manager.cc:1398] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:06.244535 6780 raft_consensus.cc:504] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.250010 6779 leader_election.cc:288] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.244268 6784 raft_consensus.cc:686] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.250710 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (table default.Table_90ae737e5e994b0bb0be30526093e6de [id=799497aabd9b4f17bc42b655e947b3ed]). Expected version 2 got 0
I20240507 08:12:06.251164 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.251358 5443 raft_consensus.cc:3037] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.253669 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.254112 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d19b7cc39a4337826819a20205aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.257030 5839 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.257310 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def4bf6a7554a4090a39b1823c4097c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.257601 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464] alter complete (version 1)
I20240507 08:12:06.258519 5839 tablet_bootstrap.cc:654] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.261094 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.261276 5444 raft_consensus.cc:2454] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.261687 5656 leader_election.cc:302] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:06.261934 6779 raft_consensus.cc:2781] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.260408 6780 leader_election.cc:288] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.262012 6779 raft_consensus.cc:482] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.262068 6779 raft_consensus.cc:3037] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.254370 6784 consensus_queue.cc:237] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.271674 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.272429 5580 raft_consensus.cc:3037] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:06.273996 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 75967.9us
I20240507 08:12:06.275187 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.276155 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ced879f35e14106b2609c78b7f24002" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.277289 5583 raft_consensus.cc:3037] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.277874 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.278578 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.281420 6780 raft_consensus.cc:482] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.281566 6780 raft_consensus.cc:504] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.282095 6780 leader_election.cc:288] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.282204 6780 raft_consensus.cc:482] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.282279 6780 raft_consensus.cc:504] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.282709 6780 leader_election.cc:288] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.283407 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.283389 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.283601 5446 raft_consensus.cc:2454] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.283700 5444 raft_consensus.cc:2454] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.284117 5656 leader_election.cc:302] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:06.284386 5656 leader_election.cc:302] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:06.284615 6553 raft_consensus.cc:2781] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.284687 6553 raft_consensus.cc:482] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.284746 6553 raft_consensus.cc:3037] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.284432 6675 raft_consensus.cc:2781] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.288094 6675 raft_consensus.cc:482] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.288182 6675 raft_consensus.cc:3037] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.288414 5843 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.288533 5843 ts_tablet_manager.cc:1367] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.248s user 0.011s sys 0.012s
I20240507 08:12:06.289181 5843 raft_consensus.cc:348] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.289350 5843 raft_consensus.cc:374] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.289395 5843 raft_consensus.cc:729] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.289592 5843 consensus_queue.cc:260] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.290059 5843 ts_tablet_manager.cc:1398] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240507 08:12:06.290285 5843 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.290944 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ce7a166001486197b78f8b02a437ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.291766 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def4bf6a7554a4090a39b1823c4097c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.297341 5843 tablet_bootstrap.cc:654] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.297546 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.300243 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_38c78dde0b4f43bca6207b3c2d2dde2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.300980 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38c78dde0b4f43bca6207b3c2d2dde2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.301951 5839 tablet_bootstrap.cc:492] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.302009 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a5fe82b3814f4ba4be0ad9369bbff3e8" } new_table_name: "default.Table_6fd5e754c95b480a8ecd9e7f593b7260" modify_external_catalogs: true
W20240507 08:12:06.302393 5654 leader_election.cc:341] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.302698 5654 leader_election.cc:341] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.302933 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.303246 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.303596 5839 ts_tablet_manager.cc:1367] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240507 08:12:06.303601 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_65d962c3598a4be5be40be25fc685388"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.304256 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65d962c3598a4be5be40be25fc685388 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.304217 5839 raft_consensus.cc:348] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.304373 5839 raft_consensus.cc:374] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.304414 5839 raft_consensus.cc:729] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.304590 5839 consensus_queue.cc:260] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.305928 5839 ts_tablet_manager.cc:1398] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240507 08:12:06.302847 5654 leader_election.cc:341] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.306192 5654 leader_election.cc:341] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.306190 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5d4ba83817d04005be08381a5c9c15b3" } modify_external_catalogs: true
I20240507 08:12:06.306406 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5d4ba83817d04005be08381a5c9c15b3" } modify_external_catalogs: true
W20240507 08:12:06.306552 5654 leader_election.cc:341] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.306831 5654 leader_election.cc:341] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.306950 5839 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.314358 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e5c1bc3385134d8fbcc94458ef0612ec
I20240507 08:12:06.315227 5840 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.315346 5840 ts_tablet_manager.cc:1367] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.208s user 0.006s sys 0.000s
I20240507 08:12:06.315351 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_91f1591433454159bdb582ad57a698d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:06.315979 5840 raft_consensus.cc:348] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:06.316069 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_91f1591433454159bdb582ad57a698d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.316144 5840 raft_consensus.cc:374] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.316195 5840 raft_consensus.cc:729] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.316484 5840 consensus_queue.cc:260] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.317111 5840 ts_tablet_manager.cc:1398] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:06.317365 5840 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:06.318312 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c40ed1afee294143ab85a8c03d080b74 consensus state has no leader
I20240507 08:12:06.320590 5256 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_bc3afc777e354e2e8a8ae7266a761d73 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677] alter complete (version 3)
I20240507 08:12:06.321949 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d2fdba650f464a74b88fe721d2987002"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.322415 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2fdba650f464a74b88fe721d2987002 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.324730 5840 tablet_bootstrap.cc:654] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.328356 6603 raft_consensus.cc:482] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.328492 6603 raft_consensus.cc:504] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.329149 6603 leader_election.cc:288] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:06.329942 5517 leader_election.cc:341] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 34bebec088a34ebbb06d2e707d54da4c
W20240507 08:12:06.330049 5520 leader_election.cc:341] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 34bebec088a34ebbb06d2e707d54da4c
I20240507 08:12:06.330150 5520 leader_election.cc:302] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.330801 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 707) took 2035 ms. Trace:
I20240507 08:12:06.330921 6603 raft_consensus.cc:2726] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.330891 6513 rpcz_store.cc:276] 0507 08:12:04.295655 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.296142 (+ 487us) service_pool.cc:225] Handling call
0507 08:12:05.734088 (+1437946us) ts_tablet_manager.cc:1120] Deleting tablet b22334378eec4ab3885d0f9d2d812a3e
0507 08:12:05.734098 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:05.937719 (+203621us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.233470 (+295751us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.330782 (+ 97312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1437789}
I20240507 08:12:06.331197 6513 tablet_replica.cc:330] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:06.331387 6513 raft_consensus.cc:2229] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.331522 6513 raft_consensus.cc:2258] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.331852 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b22334378eec4ab3885d0f9d2d812a3e" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.332024 6513 ts_tablet_manager.cc:1875] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:06.332386 5517 leader_election.cc:341] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240507 08:12:06.332470 5517 leader_election.cc:302] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.332661 6603 raft_consensus.cc:2726] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.333773 6508 raft_consensus.cc:504] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.334333 6508 leader_election.cc:288] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.334627 5443 raft_consensus.cc:2454] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.335611 6547 ts_tablet_manager.cc:1888] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.335696 6547 log.cc:1196] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5f2b43daab6b4deeac94fc6f1ff34a76
I20240507 08:12:06.335964 6547 ts_tablet_manager.cc:1909] T 5f2b43daab6b4deeac94fc6f1ff34a76 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:06.336364 5841 ts_tablet_manager.cc:1888] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:06.336483 5841 log.cc:1196] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1b19d8e28f164c588946ea5e81ec320e
I20240507 08:12:06.336797 5841 ts_tablet_manager.cc:1909] T 1b19d8e28f164c588946ea5e81ec320e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:06.340071 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 12e85df8b900468ba2584fd8f2e9bb2d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.340287 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet c7d19b7cc39a4337826819a20205aecc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.340447 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ced879f35e14106b2609c78b7f24002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.340581 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def4bf6a7554a4090a39b1823c4097c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.340714 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet d33a3ccd7a5e4b9e956735443d4c484a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.340852 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.343295 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.344798 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.345002 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.345156 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.345293 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.350394 5656 leader_election.cc:302] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.351166 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.351394 5443 raft_consensus.cc:3037] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:06.354593 5654 leader_election.cc:341] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.355881 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b22334378eec4ab3885d0f9d2d812a3e (table default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]) successfully deleted
I20240507 08:12:06.356801 6603 raft_consensus.cc:482] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.356923 6603 raft_consensus.cc:504] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.357550 6603 leader_election.cc:288] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:06.358531 5520 leader_election.cc:341] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 107d9b3f95e14044854ad0bf91a268ee
I20240507 08:12:06.358878 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.362970 6508 raft_consensus.cc:2781] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:06.364601 5656 consensus_peers.cc:485] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4fcc1d7e309946d783a524b61d5f1269. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.364629 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:06.367412 5517 leader_election.cc:341] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 107d9b3f95e14044854ad0bf91a268ee
I20240507 08:12:06.367517 5517 leader_election.cc:302] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.367787 6603 raft_consensus.cc:2726] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.369628 6508 raft_consensus.cc:686] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.371523 6508 consensus_queue.cc:237] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.374984 5840 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.375391 5840 ts_tablet_manager.cc:1367] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.058s user 0.005s sys 0.008s
I20240507 08:12:06.381973 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.382568 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6d8668c1f9b94e15b7b7603a49f11a7d" } modify_external_catalogs: true
I20240507 08:12:06.382963 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6d8668c1f9b94e15b7b7603a49f11a7d" } modify_external_catalogs: true
I20240507 08:12:06.384388 6508 consensus_queue.cc:1035] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:06.385969 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a5bc6a7273624dceb2e737e0f7df622c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:06.387248 6753 raft_consensus.cc:482] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:06.387269 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5bc6a7273624dceb2e737e0f7df622c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.387367 6753 raft_consensus.cc:504] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.388017 6753 leader_election.cc:288] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.388581 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e63c3f3ecc84913ad05864be450e9b0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.388720 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1] alter complete (version 4)
W20240507 08:12:06.389075 5378 leader_election.cc:341] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.390239 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.394987 6508 raft_consensus.cc:482] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.395174 6508 raft_consensus.cc:504] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.395974 6508 leader_election.cc:288] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.396834 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.397559 5654 leader_election.cc:341] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.398197 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.398391 5446 raft_consensus.cc:2454] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.398849 5656 leader_election.cc:302] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.399538 6508 raft_consensus.cc:2781] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.399647 6508 raft_consensus.cc:482] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.399848 6614 raft_consensus.cc:482] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.399992 6614 raft_consensus.cc:504] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.401351 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.401844 5378 leader_election.cc:341] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.403008 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.403487 5377 leader_election.cc:341] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.403580 5377 leader_election.cc:302] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.403837 6753 raft_consensus.cc:2726] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.404052 6614 leader_election.cc:288] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.399724 6508 raft_consensus.cc:3037] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.410118 6780 raft_consensus.cc:482] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.410292 6780 raft_consensus.cc:504] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.409616 5840 raft_consensus.cc:348] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.411049 6780 leader_election.cc:288] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.411123 5840 raft_consensus.cc:374] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.411934 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.412365 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:06.412562 5654 leader_election.cc:341] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.412590 5446 raft_consensus.cc:2454] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.411754 5840 raft_consensus.cc:729] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.412844 6780 raft_consensus.cc:482] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.412943 5656 leader_election.cc:302] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.412989 6780 raft_consensus.cc:504] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.413209 6784 raft_consensus.cc:2781] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.413268 6784 raft_consensus.cc:482] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.413326 6784 raft_consensus.cc:3037] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.413527 6780 leader_election.cc:288] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.414372 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.414387 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.414526 5446 raft_consensus.cc:2454] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:06.414851 5654 leader_election.cc:341] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.415081 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.415201 5656 leader_election.cc:302] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.415632 6780 raft_consensus.cc:2781] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.415702 6780 raft_consensus.cc:482] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.415750 6780 raft_consensus.cc:3037] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.419148 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.420763 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.421485 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.422235 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.422161 5840 consensus_queue.cc:260] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.426622 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 718) took 1516 ms. Trace:
I20240507 08:12:06.426931 5447 raft_consensus.cc:2454] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.427310 5656 leader_election.cc:302] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.427537 6773 raft_consensus.cc:2781] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.427650 6773 raft_consensus.cc:686] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.427865 6773 consensus_queue.cc:237] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.431453 5840 ts_tablet_manager.cc:1398] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.056s user 0.004s sys 0.000s
I20240507 08:12:06.432380 6553 raft_consensus.cc:504] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.432977 6553 leader_election.cc:288] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.435907 5840 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:06.438917 6547 rpcz_store.cc:276] 0507 08:12:04.910235 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.910349 (+ 114us) service_pool.cc:225] Handling call
0507 08:12:06.034649 (+1124300us) ts_tablet_manager.cc:1120] Deleting tablet 5f2b43daab6b4deeac94fc6f1ff34a76
0507 08:12:06.034657 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.133357 (+ 98700us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.335511 (+202154us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.426599 (+ 91088us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":19142,"tablet-delete.queue_time_us":1121681}
I20240507 08:12:06.439260 6547 tablet_replica.cc:330] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:06.439441 6547 raft_consensus.cc:2229] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.439589 6547 raft_consensus.cc:2258] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.440142 6547 ts_tablet_manager.cc:1875] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.440227 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_939d165f98ef46009d0837f15dc13dbd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.440932 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_939d165f98ef46009d0837f15dc13dbd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.440940 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:06.441465 5654 leader_election.cc:341] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.441809 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c707458d1c094d87b2204683b28b49e4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.442023 5447 raft_consensus.cc:3037] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.453893 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4cf5e458eb7e4fe4b1a953b18d982297" } new_table_name: "default.Table_d55b94afe9884d6d8b9f209875c7656d" modify_external_catalogs: true
I20240507 08:12:06.463546 5839 tablet_bootstrap.cc:654] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:06.464578 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 169415us
I20240507 08:12:06.465638 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet e5c1bc3385134d8fbcc94458ef0612ec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.466398 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 907b5cb0e7a54903abe0b0d0e8d617dc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.466869 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 973a199a533b4bbea3c2615acdbf95ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.467392 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 107d9b3f95e14044854ad0bf91a268ee with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.468362 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5f2b43daab6b4deeac94fc6f1ff34a76 (table default.Table_390a0d927d2c43f997940b8fed566a5a [id=b90f786ce1894a5e8ba53ceea2fbce9b]) successfully deleted
W20240507 08:12:06.470778 5654 consensus_peers.cc:485] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.475034 6614 raft_consensus.cc:482] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.475194 6614 raft_consensus.cc:504] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.476162 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.476253 6614 leader_election.cc:288] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:06.476645 5378 leader_election.cc:341] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.477115 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.477622 5377 leader_election.cc:341] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.477743 5377 leader_election.cc:302] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.478148 6614 raft_consensus.cc:2726] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.486373 5840 tablet_bootstrap.cc:654] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.496990 6614 raft_consensus.cc:482] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.497262 6614 raft_consensus.cc:504] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.498558 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.499169 5378 leader_election.cc:341] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.499672 6614 leader_election.cc:288] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:06.499783 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 consensus state has no leader
I20240507 08:12:06.499943 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.500360 5377 leader_election.cc:341] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.500439 5377 leader_election.cc:302] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.500839 6614 raft_consensus.cc:2726] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.508416 5840 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.508543 5840 ts_tablet_manager.cc:1367] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.000s
I20240507 08:12:06.509157 5840 raft_consensus.cc:348] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.509333 5840 raft_consensus.cc:374] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.509373 5840 raft_consensus.cc:729] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.509538 5840 consensus_queue.cc:260] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.510008 5840 ts_tablet_manager.cc:1398] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240507 08:12:06.510871 5839 tablet_bootstrap.cc:492] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.511035 5839 ts_tablet_manager.cc:1367] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.204s user 0.010s sys 0.007s
I20240507 08:12:06.511803 5839 raft_consensus.cc:348] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.511967 5839 raft_consensus.cc:374] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.512005 5839 raft_consensus.cc:729] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.512171 5839 consensus_queue.cc:260] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.513134 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
I20240507 08:12:06.513741 5839 ts_tablet_manager.cc:1398] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:06.514343 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8c20f0267d2b49e2a564ec31f3cd52a1" } modify_external_catalogs: true
I20240507 08:12:06.514573 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8c20f0267d2b49e2a564ec31f3cd52a1" } modify_external_catalogs: true
I20240507 08:12:06.515029 5839 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.515283 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9178d27018a341e2bc0fb33f4e36c267" } modify_external_catalogs: true
I20240507 08:12:06.515523 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9178d27018a341e2bc0fb33f4e36c267" } modify_external_catalogs: true
I20240507 08:12:06.516449 5839 tablet_bootstrap.cc:654] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.519521 5840 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:06.519929 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ef912d7f601f4332a3e635270c865ef9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.520474 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef912d7f601f4332a3e635270c865ef9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.525068 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_26e6e2091f7f4938ba8bdce96b06aeb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.525777 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26e6e2091f7f4938ba8bdce96b06aeb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.526861 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.527156 6675 raft_consensus.cc:504] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.527465 6779 raft_consensus.cc:504] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.527772 6675 leader_election.cc:288] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.527951 6779 leader_election.cc:288] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.531810 6513 ts_tablet_manager.cc:1888] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.531915 6513 log.cc:1196] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/98aa4ec7b93943e5bf8f9758b3bb71e0
I20240507 08:12:06.527176 5447 raft_consensus.cc:2454] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.532186 6513 ts_tablet_manager.cc:1909] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:06.536374 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.536607 5447 raft_consensus.cc:3037] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.536983 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.537169 5444 raft_consensus.cc:3037] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.538169 5840 tablet_bootstrap.cc:654] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.539739 5839 tablet_bootstrap.cc:492] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.539673 5656 leader_election.cc:302] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.539852 5839 ts_tablet_manager.cc:1367] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.001s
I20240507 08:12:06.540081 6779 raft_consensus.cc:2781] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.540191 6779 raft_consensus.cc:686] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.540434 6779 consensus_queue.cc:237] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.540521 5839 raft_consensus.cc:348] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.540683 5839 raft_consensus.cc:374] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.540732 5839 raft_consensus.cc:729] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.540937 5839 consensus_queue.cc:260] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.541936 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "070fdbf0b00b4582a184729c2f2a4b69" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:06.542007 5249 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.254:37179, user_credentials={real_user=slave}} blocked reactor thread for 44067.1us
I20240507 08:12:06.542524 5839 ts_tablet_manager.cc:1398] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:06.542766 5839 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.543201 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "302fbf120e9d472ba6bdf9e6cbdf6dbd" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:06.543593 5654 leader_election.cc:341] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.546281 5654 leader_election.cc:341] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.549319 6603 raft_consensus.cc:482] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.549436 6603 raft_consensus.cc:504] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.549983 6603 leader_election.cc:288] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.550310 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193ec0b463d04835b74df80229c85cde" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.550494 5718 raft_consensus.cc:2454] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
W20240507 08:12:06.550801 5520 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 193ec0b463d04835b74df80229c85cde
I20240507 08:12:06.550907 5517 leader_election.cc:302] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.551141 6603 raft_consensus.cc:2781] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.551208 6603 raft_consensus.cc:482] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.551276 6603 raft_consensus.cc:3037] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.555473 5839 tablet_bootstrap.cc:654] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.556830 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
I20240507 08:12:06.558434 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f294baf04bf74dfe8500cb8c1452d8b3
I20240507 08:12:06.559708 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.559863 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.559569 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dec376d239c14388b1cdd9fa9b67873f" } new_table_name: "default.Table_2e10b1940aa14c4fb9820be1553d09e5" modify_external_catalogs: true
I20240507 08:12:06.562350 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5340a70e2ea3467e844c940b4b41d9ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.563617 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5340a70e2ea3467e844c940b4b41d9ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.566090 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_53fc83cdbcd7439ba2dd0ca50cae2094" } new_table_name: "default.Table_012b7342d513481a9f0bde95ef70b498" modify_external_catalogs: true
W20240507 08:12:06.567668 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e561c811a1de48f9bc91175a3d22805a consensus state has no leader
I20240507 08:12:06.568420 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d73a8ee5a4848628ed2472bdc0b084b" } modify_external_catalogs: true
I20240507 08:12:06.568658 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d73a8ee5a4848628ed2472bdc0b084b" } modify_external_catalogs: true
I20240507 08:12:06.573112 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:06.573648 5249 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 13fa1a807d3548d6836652379d455b0d (table default.Table_64eb52ba1a3746b692ef0a3512c3eea1 [id=7e854bf3351b4493a16b7bd7aa685baf], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 13fa1a807d3548d6836652379d455b0d consensus state has no leader
I20240507 08:12:06.589051 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b8da67c846954c4a95c5a20759624d6f" } new_table_name: "default.Table_bb1df10cd5d54d23bb3720be57fd9a46" modify_external_catalogs: true
I20240507 08:12:06.593472 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ef7bea1804634eaf8efbf1bc57d940bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.594488 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef7bea1804634eaf8efbf1bc57d940bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.599766 5583 raft_consensus.cc:3037] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:06.603932 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c56bd481b4d4c8285e70d9b01f9e8e0 consensus state has no leader
W20240507 08:12:06.604586 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d432a59bdd344ddaaf710de73b543e6 consensus state has no leader
W20240507 08:12:06.610469 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aafd1dc2eaf84390b072f01a9e76232d consensus state has no leader
I20240507 08:12:06.618636 5840 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.618857 5840 ts_tablet_manager.cc:1367] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.099s user 0.011s sys 0.005s
I20240507 08:12:06.619586 5840 raft_consensus.cc:348] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.619940 6614 raft_consensus.cc:482] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.620091 6614 raft_consensus.cc:504] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.620728 6614 leader_election.cc:288] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.621408 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.622094 5378 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.621749 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.622681 5377 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.622771 5377 leader_election.cc:302] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.623162 6614 raft_consensus.cc:2726] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.623356 5840 raft_consensus.cc:374] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.623473 5840 raft_consensus.cc:729] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.623724 5840 consensus_queue.cc:260] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.626595 6508 raft_consensus.cc:504] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.627137 6508 leader_election.cc:288] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.627434 5843 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.627557 5843 ts_tablet_manager.cc:1367] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.337s user 0.103s sys 0.122s
I20240507 08:12:06.628299 5843 raft_consensus.cc:348] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.628487 5843 raft_consensus.cc:374] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.628538 5843 raft_consensus.cc:729] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.628721 5843 consensus_queue.cc:260] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.629694 6784 raft_consensus.cc:504] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.632442 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 709) took 2336 ms. Trace:
I20240507 08:12:06.632539 6513 rpcz_store.cc:276] 0507 08:12:04.295849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.296409 (+ 560us) service_pool.cc:225] Handling call
0507 08:12:06.331149 (+2034740us) ts_tablet_manager.cc:1120] Deleting tablet 98aa4ec7b93943e5bf8f9758b3bb71e0
0507 08:12:06.331159 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.433221 (+102062us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.531694 (+ 98473us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.632422 (+100728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034449}
I20240507 08:12:06.632926 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 711) took 2295 ms. Trace:
I20240507 08:12:06.632993 6513 rpcz_store.cc:276] 0507 08:12:04.337870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.337925 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:06.632818 (+2294893us) ts_tablet_manager.cc:1120] Deleting tablet 07a9324570984372af7b9574f0cf455a
0507 08:12:06.632826 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.632912 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2294660}
I20240507 08:12:06.633118 5843 ts_tablet_manager.cc:1398] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:06.633260 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 712) took 2288 ms. Trace:
I20240507 08:12:06.633318 6513 rpcz_store.cc:276] 0507 08:12:04.344447 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.344500 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:06.633176 (+2288676us) ts_tablet_manager.cc:1120] Deleting tablet 85dc58133d2b4f5496b4e570ae6494c0
0507 08:12:06.633183 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.633247 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2288469}
I20240507 08:12:06.633394 5843 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.633558 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 714) took 2272 ms. Trace:
I20240507 08:12:06.633646 6513 rpcz_store.cc:276] 0507 08:12:04.361117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.361174 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:06.633482 (+2272308us) ts_tablet_manager.cc:1120] Deleting tablet 039dcaf121b94ebab94ec522346d9d82
0507 08:12:06.633490 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.633547 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2272074}
I20240507 08:12:06.633860 6513 tablet_replica.cc:330] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:06.634014 6513 raft_consensus.cc:2229] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.634152 6513 raft_consensus.cc:2258] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.634665 6513 ts_tablet_manager.cc:1875] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.634956 5843 tablet_bootstrap.cc:654] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.636427 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.636857 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.637619 6547 ts_tablet_manager.cc:1888] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.637725 6547 log.cc:1196] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/88a2cfb5f046434c8340234a447a7e65
I20240507 08:12:06.637706 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.637871 5446 raft_consensus.cc:3037] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.638021 6547 ts_tablet_manager.cc:1909] T 88a2cfb5f046434c8340234a447a7e65 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:06.638244 5444 raft_consensus.cc:2454] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.638584 6603 raft_consensus.cc:504] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.639219 6603 leader_election.cc:288] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.640245 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "193ec0b463d04835b74df80229c85cde" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:06.640319 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.640406 5718 raft_consensus.cc:3037] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.640460 5445 raft_consensus.cc:3037] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:06.640980 5654 leader_election.cc:341] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.641146 5654 leader_election.cc:341] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.646643 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 720) took 2250 ms. Trace:
I20240507 08:12:06.646724 5841 rpcz_store.cc:276] 0507 08:12:04.396467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.397679 (+ 1212us) service_pool.cc:225] Handling call
0507 08:12:06.210102 (+1812423us) ts_tablet_manager.cc:1120] Deleting tablet 1b19d8e28f164c588946ea5e81ec320e
0507 08:12:06.210111 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.293865 (+ 83754us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.336267 (+ 42402us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.646624 (+310357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1812262}
I20240507 08:12:06.647063 5841 tablet_replica.cc:330] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:06.647228 5841 raft_consensus.cc:2229] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.647373 5841 raft_consensus.cc:2258] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.399699 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e63c3f3ecc84913ad05864be450e9b0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.647707 5720 raft_consensus.cc:2454] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:06.648157 5377 leader_election.cc:302] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.648788 6753 raft_consensus.cc:2781] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.648854 6753 raft_consensus.cc:482] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.648905 6753 raft_consensus.cc:3037] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.649737 6784 leader_election.cc:288] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.650460 5840 ts_tablet_manager.cc:1398] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240507 08:12:06.653517 6684 raft_consensus.cc:482] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.653648 6684 raft_consensus.cc:504] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.654187 6684 leader_election.cc:288] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.654696 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.655125 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1b19d8e28f164c588946ea5e81ec320e (table default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]) successfully deleted
W20240507 08:12:06.655203 5378 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.655804 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:06.640136 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]) successfully deleted
W20240507 08:12:06.657052 5377 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.657140 5377 leader_election.cc:302] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:06.657207 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 07a9324570984372af7b9574f0cf455a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 07a9324570984372af7b9574f0cf455a already in progress: creating tablet
W20240507 08:12:06.657420 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 85dc58133d2b4f5496b4e570ae6494c0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 85dc58133d2b4f5496b4e570ae6494c0 already in progress: creating tablet
I20240507 08:12:06.657475 6684 raft_consensus.cc:2726] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:06.658507 5654 consensus_peers.cc:485] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.659063 5840 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:06.659490 5656 leader_election.cc:302] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.659724 6784 raft_consensus.cc:2781] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.659847 6784 raft_consensus.cc:686] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.660178 6784 consensus_queue.cc:237] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.661108 6603 raft_consensus.cc:482] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.661249 6603 raft_consensus.cc:504] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:06.662123 5520 leader_election.cc:341] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 193ec0b463d04835b74df80229c85cde
I20240507 08:12:06.662653 6603 leader_election.cc:288] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.662950 5843 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:06.663050 5377 consensus_peers.cc:485] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98aa4ec7b93943e5bf8f9758b3bb71e0. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.663082 5843 ts_tablet_manager.cc:1367] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
W20240507 08:12:06.663175 5517 leader_election.cc:341] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 907b5cb0e7a54903abe0b0d0e8d617dc
W20240507 08:12:06.663729 5520 leader_election.cc:341] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:12:06.663761 5843 raft_consensus.cc:348] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.663923 5843 raft_consensus.cc:374] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.663967 5843 raft_consensus.cc:729] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.663919 5520 leader_election.cc:302] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.664165 5843 consensus_queue.cc:260] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.664600 5843 ts_tablet_manager.cc:1398] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240507 08:12:06.664783 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 039dcaf121b94ebab94ec522346d9d82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 039dcaf121b94ebab94ec522346d9d82 already in progress: creating tablet
I20240507 08:12:06.664856 5843 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.665544 5841 ts_tablet_manager.cc:1875] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.665787 5839 tablet_bootstrap.cc:492] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:06.665911 5839 ts_tablet_manager.cc:1367] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.123s user 0.005s sys 0.012s
I20240507 08:12:06.666407 5843 tablet_bootstrap.cc:654] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.669706 6603 raft_consensus.cc:2726] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.671041 5840 tablet_bootstrap.cc:654] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.671640 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet aa485207d0f140c99a9386cb4002c480 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:06.671610 5839 raft_consensus.cc:348] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.671797 5839 raft_consensus.cc:374] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.671864 5839 raft_consensus.cc:729] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.672076 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.673439 5839 consensus_queue.cc:260] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.677488 5839 ts_tablet_manager.cc:1398] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:06.678544 5839 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:06.683226 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.692642 5840 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:06.692770 5840 ts_tablet_manager.cc:1367] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20240507 08:12:06.694525 5840 raft_consensus.cc:348] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.694705 5840 raft_consensus.cc:374] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.694746 5840 raft_consensus.cc:729] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.694923 5840 consensus_queue.cc:260] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.695175 5839 tablet_bootstrap.cc:654] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.698266 6508 consensus_queue.cc:1035] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:06.702248 5840 ts_tablet_manager.cc:1398] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:06.702622 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c083a659194342cdbdbb42898c701c6d" } new_table_name: "default.Table_c1cc14a610bb4d60a5ad3f51eeb4aeee" modify_external_catalogs: true
I20240507 08:12:06.704131 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 150efae7834740b798af86ae1dcfe445
I20240507 08:12:06.705196 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 150efae7834740b798af86ae1dcfe445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.706255 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 150efae7834740b798af86ae1dcfe445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.707378 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 150efae7834740b798af86ae1dcfe445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.707423 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6f373518a45b48f8872daf99361fe15b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.707937 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f373518a45b48f8872daf99361fe15b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.708379 5840 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:06.709491 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7e11a37eccea4b7ca542f722051823d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:06.709662 5840 tablet_bootstrap.cc:654] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.712245 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3f826c69443f463ca9524bf8917815ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.712725 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f826c69443f463ca9524bf8917815ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.714224 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 242ea35c61404a91b1211c5ad9207728
I20240507 08:12:06.714350 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fdd7bcf579774a45a7fe204ceec43195"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.714797 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdd7bcf579774a45a7fe204ceec43195 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.715822 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:06.716037 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e11a37eccea4b7ca542f722051823d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.716264 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.716542 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.720414 6684 raft_consensus.cc:482] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.720583 6684 raft_consensus.cc:504] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.721355 6684 leader_election.cc:288] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:06.723821 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:06.724684 6780 raft_consensus.cc:504] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.725188 6780 leader_election.cc:288] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.727593 5718 raft_consensus.cc:2454] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:12:06.728391 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:06.728816 5654 leader_election.cc:341] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.729151 5447 raft_consensus.cc:2454] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.737365 5517 leader_election.cc:302] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.737730 6603 raft_consensus.cc:2781] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.737888 6603 raft_consensus.cc:686] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:12:06.737988 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 4fcc1d7e309946d783a524b61d5f1269 with delete_type TABLET_DATA_DELETED (replaced by tablet 59dbc2eaa45d4da58951b45423f284fd) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.738162 6603 consensus_queue.cc:237] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:06.738457 5378 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.741389 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 719) took 1818 ms. Trace:
I20240507 08:12:06.741500 6547 rpcz_store.cc:276] 0507 08:12:04.922664 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.923492 (+ 828us) service_pool.cc:225] Handling call
0507 08:12:06.439212 (+1515720us) ts_tablet_manager.cc:1120] Deleting tablet 88a2cfb5f046434c8340234a447a7e65
0507 08:12:06.439222 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.532467 (+ 93245us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.637501 (+105034us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.741335 (+103834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515535}
I20240507 08:12:06.742115 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 725) took 1742 ms. Trace:
I20240507 08:12:06.742128 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 88a2cfb5f046434c8340234a447a7e65 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
I20240507 08:12:06.742216 6547 rpcz_store.cc:276] 0507 08:12:04.999381 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.999667 (+ 286us) service_pool.cc:225] Handling call
0507 08:12:06.741913 (+1742246us) ts_tablet_manager.cc:1120] Deleting tablet 4e3ea4658bb946439e4a50962444b3e2
0507 08:12:06.741925 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.742076 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742014}
I20240507 08:12:06.742630 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 721) took 1743 ms. Trace:
W20240507 08:12:06.742738 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4e3ea4658bb946439e4a50962444b3e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3ea4658bb946439e4a50962444b3e2 already in progress: creating tablet
I20240507 08:12:06.742722 6547 rpcz_store.cc:276] 0507 08:12:04.999062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.004636 (+ 5574us) service_pool.cc:225] Handling call
0507 08:12:06.742485 (+1737849us) ts_tablet_manager.cc:1120] Deleting tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5
0507 08:12:06.742495 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.742575 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737665}
I20240507 08:12:06.743094 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 722) took 1743 ms. Trace:
I20240507 08:12:06.743192 6547 rpcz_store.cc:276] 0507 08:12:04.999152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.004857 (+ 5705us) service_pool.cc:225] Handling call
0507 08:12:06.742964 (+1738107us) ts_tablet_manager.cc:1120] Deleting tablet 96ef77ef4f474107bf31395f5f30c12f
0507 08:12:06.742974 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.743055 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1737948}
I20240507 08:12:06.743530 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 723) took 1744 ms. Trace:
I20240507 08:12:06.743629 6547 rpcz_store.cc:276] 0507 08:12:04.999236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.005040 (+ 5804us) service_pool.cc:225] Handling call
0507 08:12:06.743413 (+1738373us) ts_tablet_manager.cc:1120] Deleting tablet 8a397c40278141b7bb80ea5fcdbb125b
0507 08:12:06.743423 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.743497 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738227}
W20240507 08:12:06.743870 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 already in progress: creating tablet
I20240507 08:12:06.744011 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 724) took 1744 ms. Trace:
W20240507 08:12:06.744176 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 96ef77ef4f474107bf31395f5f30c12f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96ef77ef4f474107bf31395f5f30c12f already in progress: creating tablet
I20240507 08:12:06.744110 6547 rpcz_store.cc:276] 0507 08:12:04.999309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.005226 (+ 5917us) service_pool.cc:225] Handling call
0507 08:12:06.743887 (+1738661us) ts_tablet_manager.cc:1120] Deleting tablet ed0018356e254b82995061ef7290556d
0507 08:12:06.743897 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.743970 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1738508}
W20240507 08:12:06.745842 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8a397c40278141b7bb80ea5fcdbb125b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a397c40278141b7bb80ea5fcdbb125b already in progress: creating tablet
W20240507 08:12:06.746125 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ed0018356e254b82995061ef7290556d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0018356e254b82995061ef7290556d already in progress: creating tablet
I20240507 08:12:06.747346 6780 consensus_queue.cc:1035] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:06.750030 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62b8fb67327b4f779f3bb494c03616a3 consensus state has no leader
I20240507 08:12:06.750211 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81a1a42a22704974b1ef6cbd17d91854" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.750430 5447 raft_consensus.cc:3037] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.752247 5656 leader_election.cc:302] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.752537 6780 raft_consensus.cc:2781] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.752636 6780 raft_consensus.cc:686] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.752861 6780 consensus_queue.cc:237] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.754005 6779 consensus_queue.cc:1035] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:06.756731 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eb79af3080394424a223e89504916751"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.757468 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb79af3080394424a223e89504916751 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.760296 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aa2b432cc7ef4604b737ade9696fe60f" } modify_external_catalogs: true
I20240507 08:12:06.760385 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 742) took 1419 ms. Trace:
I20240507 08:12:06.760533 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_aa2b432cc7ef4604b737ade9696fe60f" } modify_external_catalogs: true
I20240507 08:12:06.760502 6547 rpcz_store.cc:276] 0507 08:12:05.325838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.326473 (+ 635us) service_pool.cc:225] Handling call
0507 08:12:06.745442 (+1418969us) ts_tablet_manager.cc:1120] Deleting tablet 3769eb670ff442e2ad2fa5b6a934512a
0507 08:12:06.745453 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.745536 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418750}
W20240507 08:12:06.761498 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3769eb670ff442e2ad2fa5b6a934512a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3769eb670ff442e2ad2fa5b6a934512a already in progress: creating tablet
I20240507 08:12:06.761549 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 746) took 1419 ms. Trace:
I20240507 08:12:06.761824 6547 rpcz_store.cc:276] 0507 08:12:05.341504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.341557 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:06.760737 (+1419180us) ts_tablet_manager.cc:1120] Deleting tablet 2bb648fe98e74248b5cd65603de58a02
0507 08:12:06.760744 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.760836 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418959}
I20240507 08:12:06.762454 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 744) took 1421 ms. Trace:
W20240507 08:12:06.762557 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2bb648fe98e74248b5cd65603de58a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb648fe98e74248b5cd65603de58a02 already in progress: creating tablet
I20240507 08:12:06.763793 6547 rpcz_store.cc:276] 0507 08:12:05.341274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.343315 (+ 2041us) service_pool.cc:225] Handling call
0507 08:12:06.762323 (+1419008us) ts_tablet_manager.cc:1120] Deleting tablet 4ad95383487e458ca162038f6940271b
0507 08:12:06.762332 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.762410 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1418829}
I20240507 08:12:06.764472 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 745) took 1423 ms. Trace:
W20240507 08:12:06.764600 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4ad95383487e458ca162038f6940271b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ad95383487e458ca162038f6940271b already in progress: creating tablet
I20240507 08:12:06.764894 6547 rpcz_store.cc:276] 0507 08:12:05.341378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.343526 (+ 2148us) service_pool.cc:225] Handling call
0507 08:12:06.764351 (+1420825us) ts_tablet_manager.cc:1120] Deleting tablet b6ba3397e33b4cc7805a25e7535a383a
0507 08:12:06.764359 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.764430 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1420695}
I20240507 08:12:06.765753 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 743) took 1424 ms. Trace:
W20240507 08:12:06.765873 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b6ba3397e33b4cc7805a25e7535a383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ba3397e33b4cc7805a25e7535a383a already in progress: creating tablet
I20240507 08:12:06.766292 6547 rpcz_store.cc:276] 0507 08:12:05.341144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.350099 (+ 8955us) service_pool.cc:225] Handling call
0507 08:12:06.765587 (+1415488us) ts_tablet_manager.cc:1120] Deleting tablet 055a30bfc61b4ff2be9f2418ca59b030
0507 08:12:06.765598 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.765684 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1415217}
I20240507 08:12:06.766669 6547 tablet_replica.cc:330] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:06.766893 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 055a30bfc61b4ff2be9f2418ca59b030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 055a30bfc61b4ff2be9f2418ca59b030 already in progress: creating tablet
I20240507 08:12:06.766927 6547 raft_consensus.cc:2229] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:06.767113 6547 raft_consensus.cc:2258] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.767740 6547 ts_tablet_manager.cc:1875] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.769028 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3d66e54ff28e4b2799cb6391b6aba67f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.769552 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d66e54ff28e4b2799cb6391b6aba67f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.773218 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_90ae737e5e994b0bb0be30526093e6de" } new_table_name: "default.Table_87bb488658cd4819a51ff1ff0993ceab" modify_external_catalogs: true
I20240507 08:12:06.775007 6808 raft_consensus.cc:482] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.775157 6808 raft_consensus.cc:504] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.775889 6808 leader_election.cc:288] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:06.777216 5520 leader_election.cc:341] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 973a199a533b4bbea3c2615acdbf95ca
W20240507 08:12:06.778370 5517 leader_election.cc:341] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 973a199a533b4bbea3c2615acdbf95ca
I20240507 08:12:06.778514 5517 leader_election.cc:302] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:06.778789 6808 raft_consensus.cc:2726] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:06.781037 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:06.782253 5377 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.782426 5377 leader_election.cc:302] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.782923 6614 raft_consensus.cc:2726] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:06.784709 5378 consensus_peers.cc:485] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.791958 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 5089db17ca904a04878d83209001c54e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.792548 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cd02a49359743c4b4e540ab0338d8ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.797216 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_77123b3ab92f4c93a46d201d8f0eb9ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.798955 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77123b3ab92f4c93a46d201d8f0eb9ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.799695 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9d3bcb49239048a198510a548f153eaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.800155 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9d3bcb49239048a198510a548f153eaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.808065 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5eee064314f4431b951651d05ed1e13" } new_table_name: "default.Table_eec3650937f2445eabf79a1c3935514b" modify_external_catalogs: true
I20240507 08:12:06.814925 5262 catalog_manager.cc:5484] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:06.815552 5262 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:06.816001 5262 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:06.819077 5698 tablet_service.cc:1467] Processing CreateTablet for tablet cd1275304cb64f2fb1e52f3d7473f119 (DEFAULT_TABLE table=default.Table_801bce464efb4d0781fea9659af5f26c [id=203605692bb347c0a6d7fc6e35c088fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.824816 5841 ts_tablet_manager.cc:1888] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:06.824971 5841 log.cc:1196] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c48da19c9b9e4207b267137d240adc5b
I20240507 08:12:06.825377 5841 ts_tablet_manager.cc:1909] T c48da19c9b9e4207b267137d240adc5b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:06.825667 5445 raft_consensus.cc:2454] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.826227 6513 ts_tablet_manager.cc:1888] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:06.826349 6513 log.cc:1196] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1b19d8e28f164c588946ea5e81ec320e
I20240507 08:12:06.826711 6513 ts_tablet_manager.cc:1909] T 1b19d8e28f164c588946ea5e81ec320e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:06.832484 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1275304cb64f2fb1e52f3d7473f119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.838332 5656 leader_election.cc:302] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.838742 6784 raft_consensus.cc:2781] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.838866 6784 raft_consensus.cc:686] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.838848 5559 tablet_service.cc:1467] Processing CreateTablet for tablet cd1275304cb64f2fb1e52f3d7473f119 (DEFAULT_TABLE table=default.Table_801bce464efb4d0781fea9659af5f26c [id=203605692bb347c0a6d7fc6e35c088fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.839074 6784 consensus_queue.cc:237] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.839373 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1275304cb64f2fb1e52f3d7473f119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.840209 6780 raft_consensus.cc:482] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.840340 6780 raft_consensus.cc:504] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.840902 6780 leader_election.cc:288] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.845295 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.846305 5654 leader_election.cc:341] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.847203 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.847412 5445 raft_consensus.cc:2454] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.847877 5656 leader_election.cc:302] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.848222 6779 raft_consensus.cc:2781] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.848287 6779 raft_consensus.cc:482] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.848341 6779 raft_consensus.cc:3037] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.851452 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fe8a85dddcfb41a18221367878c0dac7" } new_table_name: "default.Table_fec60780cac3409fb0c1d6d61f8a8e25" modify_external_catalogs: true
I20240507 08:12:06.856617 6784 raft_consensus.cc:482] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.856750 6784 raft_consensus.cc:504] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.857301 6784 leader_election.cc:288] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.858055 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.858569 5654 leader_election.cc:341] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.859194 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.859391 5445 raft_consensus.cc:2454] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.859915 5656 leader_election.cc:302] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:06.860216 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d6f8f4205f1f48abaf4fcd7113ad44a8 consensus state has no leader
I20240507 08:12:06.860306 6784 raft_consensus.cc:2781] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.860379 6784 raft_consensus.cc:482] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.860428 6784 raft_consensus.cc:3037] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.865496 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f77c7b9058644cdd88fe3ee72b888872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:06.866111 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f77c7b9058644cdd88fe3ee72b888872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:06.867878 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 96d7d8bcb635416693395b1f9ad27e2f
I20240507 08:12:06.868606 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_64eb52ba1a3746b692ef0a3512c3eea1" } new_table_name: "default.Table_69d9a0c148964285a59fc90ed39304dd" modify_external_catalogs: true
I20240507 08:12:06.870194 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:06.871424 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:06.872558 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 96d7d8bcb635416693395b1f9ad27e2f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.875989 5409 tablet_service.cc:1467] Processing CreateTablet for tablet cd1275304cb64f2fb1e52f3d7473f119 (DEFAULT_TABLE table=default.Table_801bce464efb4d0781fea9659af5f26c [id=203605692bb347c0a6d7fc6e35c088fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.876539 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd1275304cb64f2fb1e52f3d7473f119. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.882872 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 2572985bc2444dc7b051f8fe008e3ad4 (DEFAULT_TABLE table=default.Table_215f0b92297748bd926f9b40fea2f149 [id=3b28b0b34e044bdfbe5bc7b6743d081e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.901696 6675 raft_consensus.cc:482] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:06.901899 6675 raft_consensus.cc:504] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.902734 6675 leader_election.cc:288] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.903708 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:06.904326 5654 leader_election.cc:341] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:06.904553 5654 consensus_peers.cc:485] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.905071 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:06.905247 5445 raft_consensus.cc:2454] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:06.905656 5656 leader_election.cc:302] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.906208 6675 raft_consensus.cc:2781] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:06.906303 6675 raft_consensus.cc:482] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:06.906392 6675 raft_consensus.cc:3037] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.913853 6684 consensus_queue.cc:1035] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:06.920575 6753 raft_consensus.cc:504] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.921114 6753 leader_election.cc:288] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:06.921404 5443 consensus_meta.cc:287] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Time spent flushing consensus metadata: real 0.570s user 0.002s sys 0.000s
I20240507 08:12:06.921471 5443 raft_consensus.cc:2454] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:06.922122 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e63c3f3ecc84913ad05864be450e9b0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:06.922293 5720 raft_consensus.cc:3037] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:06.922737 5378 consensus_peers.cc:485] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:06.922921 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e63c3f3ecc84913ad05864be450e9b0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:06.923354 5378 leader_election.cc:341] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.927335 6779 raft_consensus.cc:504] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.927922 6779 leader_election.cc:288] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:06.928342 6547 ts_tablet_manager.cc:1888] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:06.928459 6547 log.cc:1196] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/916e33e40ef44d06a359b82d8fe48157
I20240507 08:12:06.928824 6547 ts_tablet_manager.cc:1909] T 916e33e40ef44d06a359b82d8fe48157 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:06.934895 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2572985bc2444dc7b051f8fe008e3ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.937966 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:06.938153 5843 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.938279 5843 ts_tablet_manager.cc:1367] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.273s user 0.072s sys 0.140s
I20240507 08:12:06.939029 5843 raft_consensus.cc:348] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.939206 5843 raft_consensus.cc:374] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.939252 5843 raft_consensus.cc:729] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.939426 5843 consensus_queue.cc:260] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:06.940052 5654 leader_election.cc:341] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:06.940081 5843 ts_tablet_manager.cc:1398] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:06.940454 5843 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.920276 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 715) took 2559 ms. Trace:
I20240507 08:12:06.941843 5843 tablet_bootstrap.cc:654] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.941819 6513 rpcz_store.cc:276] 0507 08:12:04.361230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.361855 (+ 625us) service_pool.cc:225] Handling call
0507 08:12:06.633822 (+2271967us) ts_tablet_manager.cc:1120] Deleting tablet 1b19d8e28f164c588946ea5e81ec320e
0507 08:12:06.633830 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.733182 (+ 99352us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.826107 (+ 92925us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.920253 (+ 94146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2271766}
I20240507 08:12:06.942198 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 713) took 2581 ms. Trace:
I20240507 08:12:06.942261 6513 rpcz_store.cc:276] 0507 08:12:04.361013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.362221 (+ 1208us) service_pool.cc:225] Handling call
0507 08:12:06.942074 (+2579853us) ts_tablet_manager.cc:1120] Deleting tablet 8e1f91718260466d99acfc9892dea536
0507 08:12:06.942083 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.942183 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2579669}
I20240507 08:12:06.942476 6513 tablet_replica.cc:330] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:06.942629 6513 raft_consensus.cc:2229] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:06.942912 6513 pending_rounds.cc:62] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:06.942953 6513 pending_rounds.cc:68] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924772738322432 op_type: NO_OP noop_request { }
I20240507 08:12:06.943035 6513 raft_consensus.cc:2866] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:06.943099 6513 raft_consensus.cc:2258] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:06.943580 6513 ts_tablet_manager.cc:1875] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:06.947932 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:06.948102 5443 raft_consensus.cc:3037] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:06.948760 5656 leader_election.cc:302] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:06.949147 6508 raft_consensus.cc:2781] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:06.949244 6508 raft_consensus.cc:686] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:06.949506 5843 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.949446 6508 consensus_queue.cc:237] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.949661 5843 ts_tablet_manager.cc:1367] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20240507 08:12:06.950244 5843 raft_consensus.cc:348] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.950518 5843 raft_consensus.cc:374] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.950573 5843 raft_consensus.cc:729] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.950786 5843 consensus_queue.cc:260] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.951354 5843 ts_tablet_manager.cc:1398] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:06.951599 5843 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.953437 5843 tablet_bootstrap.cc:654] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.956946 5843 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.957072 5843 ts_tablet_manager.cc:1367] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240507 08:12:06.957670 5843 raft_consensus.cc:348] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.957836 5843 raft_consensus.cc:374] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.957880 5843 raft_consensus.cc:729] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.958073 5843 consensus_queue.cc:260] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.958637 5843 ts_tablet_manager.cc:1398] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240507 08:12:06.958863 5843 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.960294 5843 tablet_bootstrap.cc:654] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.967948 5843 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:06.968072 5843 ts_tablet_manager.cc:1367] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240507 08:12:06.968647 5843 raft_consensus.cc:348] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.968828 5843 raft_consensus.cc:374] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:06.968884 5843 raft_consensus.cc:729] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:06.969122 5843 consensus_queue.cc:260] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:06.969877 5843 ts_tablet_manager.cc:1398] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:06.970170 5843 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:06.971469 5843 tablet_bootstrap.cc:654] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:06.988183 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet abd47fb41bd14411bb44cc0f25e8cac4
I20240507 08:12:06.988979 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:06.991778 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 2572985bc2444dc7b051f8fe008e3ad4 (DEFAULT_TABLE table=default.Table_215f0b92297748bd926f9b40fea2f149 [id=3b28b0b34e044bdfbe5bc7b6743d081e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.992045 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 2572985bc2444dc7b051f8fe008e3ad4 (DEFAULT_TABLE table=default.Table_215f0b92297748bd926f9b40fea2f149 [id=3b28b0b34e044bdfbe5bc7b6743d081e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.992305 5420 tablet_service.cc:1467] Processing CreateTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 (DEFAULT_TABLE table=default.Table_9178d27018a341e2bc0fb33f4e36c267 [id=014cef78aee24cd08153b4596d7e5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.992414 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2572985bc2444dc7b051f8fe008e3ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.992579 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd47fb41bd14411bb44cc0f25e8cac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.995599 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 46cc8597ca5042ca8e85d99803d07206 (DEFAULT_TABLE table=default.Table_2e80ae4bf74945cc82095e3a11e1939b [id=60900a70a269419bb8405f5c0ca13021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.995949 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cc8597ca5042ca8e85d99803d07206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.996773 5554 tablet_service.cc:1467] Processing CreateTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 (DEFAULT_TABLE table=default.Table_9178d27018a341e2bc0fb33f4e36c267 [id=014cef78aee24cd08153b4596d7e5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:06.997259 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd47fb41bd14411bb44cc0f25e8cac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:06.995663 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.000766 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 46cc8597ca5042ca8e85d99803d07206 (DEFAULT_TABLE table=default.Table_2e80ae4bf74945cc82095e3a11e1939b [id=60900a70a269419bb8405f5c0ca13021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.001260 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cc8597ca5042ca8e85d99803d07206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.017287 5840 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:07.017462 5840 ts_tablet_manager.cc:1367] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.309s user 0.008s sys 0.023s
I20240507 08:12:07.018460 5840 raft_consensus.cc:348] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.018738 5840 raft_consensus.cc:374] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.018788 5840 raft_consensus.cc:729] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.019083 5840 consensus_queue.cc:260] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.019776 5840 ts_tablet_manager.cc:1398] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:07.028252 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.028285 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1b19d8e28f164c588946ea5e81ec320e (table default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]) successfully deleted
I20240507 08:12:07.027885 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 748) took 1609 ms. Trace:
W20240507 08:12:07.028558 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8e1f91718260466d99acfc9892dea536 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8e1f91718260466d99acfc9892dea536 already in progress: creating tablet
I20240507 08:12:07.028491 6547 rpcz_store.cc:276] 0507 08:12:05.418065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.418122 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:06.766555 (+1348433us) ts_tablet_manager.cc:1120] Deleting tablet 916e33e40ef44d06a359b82d8fe48157
0507 08:12:06.766567 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.827095 (+ 60528us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.928192 (+101097us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.027844 (+ 99652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1348174}
I20240507 08:12:07.028918 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 749) took 1610 ms. Trace:
I20240507 08:12:07.029036 6547 rpcz_store.cc:276] 0507 08:12:05.418204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.419647 (+ 1443us) service_pool.cc:225] Handling call
0507 08:12:07.028752 (+1609105us) ts_tablet_manager.cc:1120] Deleting tablet 93c68ddcfa7c40c180b6b12fabe165d6
0507 08:12:07.028774 (+ 22us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.028880 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1608903}
I20240507 08:12:07.029351 6547 tablet_replica.cc:330] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:07.029304 5690 tablet_service.cc:1467] Processing CreateTablet for tablet abd47fb41bd14411bb44cc0f25e8cac4 (DEFAULT_TABLE table=default.Table_9178d27018a341e2bc0fb33f4e36c267 [id=014cef78aee24cd08153b4596d7e5c50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.029539 6816 raft_consensus.cc:482] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.029623 6547 raft_consensus.cc:2229] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.029783 6547 raft_consensus.cc:2258] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.029688 6816 raft_consensus.cc:504] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.030328 6816 leader_election.cc:288] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.030423 6547 ts_tablet_manager.cc:1875] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.030684 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 46cc8597ca5042ca8e85d99803d07206 (DEFAULT_TABLE table=default.Table_2e80ae4bf74945cc82095e3a11e1939b [id=60900a70a269419bb8405f5c0ca13021]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.031075 5447 raft_consensus.cc:2454] T 81a1a42a22704974b1ef6cbd17d91854 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:07.031431 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2572985bc2444dc7b051f8fe008e3ad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.031381 6784 raft_consensus.cc:504] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.031930 6784 leader_election.cc:288] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.034050 5656 leader_election.cc:302] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.034466 6784 raft_consensus.cc:2781] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.034571 6784 raft_consensus.cc:686] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:07.034770 6784 consensus_queue.cc:237] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.036357 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:07.036535 5447 raft_consensus.cc:3037] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.037259 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7053cfaea4a54a402c9d826613c05" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:07.037881 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 916e33e40ef44d06a359b82d8fe48157 (table default.Table_6ad0c511f95b4b09abad3bc4a13d1238 [id=14589abdd357409cbe9837a33208e00d]) successfully deleted
W20240507 08:12:07.038151 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 93c68ddcfa7c40c180b6b12fabe165d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c68ddcfa7c40c180b6b12fabe165d6 already in progress: creating tablet
W20240507 08:12:07.039870 5378 leader_election.cc:341] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.040808 5700 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a077be5574cb8bc7a2347c6d3af00 (DEFAULT_TABLE table=default.Table_b0948279632f4c32a83404acd89ae917 [id=e62c912343e24cd4bddbaba00fb6ab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.040794 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0118ef28b24641d68cf24ce9df966243" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:07.041975 5654 leader_election.cc:341] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.043896 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aa485207d0f140c99a9386cb4002c480
I20240507 08:12:07.044346 5255 catalog_manager.cc:6554] ReplaceTablet: tablet aa485207d0f140c99a9386cb4002c480 deleted and replaced by tablet 30ee9be0851243738b13c2231b554da8
I20240507 08:12:07.046690 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 3954a4e0d3104161aad427eb2781d55d (DEFAULT_TABLE table=default.Table_558ec8354d9943f6ad3e2be4108aefc6 [id=21dd3a734fb64db0bc7b318d846fbf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.047082 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3954a4e0d3104161aad427eb2781d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.047055 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 9501f5041dab4ca0afa406884f6c4aa1 (DEFAULT_TABLE table=default.Table_c3cca313bfc24efcacf2454a66c6e114 [id=345a522dce9445b58243d3d609349ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.047333 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501f5041dab4ca0afa406884f6c4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.047557 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 1f748a59b4a347e288d534ef1296d72c (DEFAULT_TABLE table=default.Table_3c747008c670486b8d63c755d6ca129b [id=e1a39de1f5c4495b9d150f186583042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.047847 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f748a59b4a347e288d534ef1296d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.049494 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88421118434749abf4126b08b115e5 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.049670 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet aa485207d0f140c99a9386cb4002c480 with delete_type TABLET_DATA_DELETED (replaced by tablet 30ee9be0851243738b13c2231b554da8) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.049856 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88421118434749abf4126b08b115e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.049994 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6cba8aeb36441d8df42b91c38d6ab3 (DEFAULT_TABLE table=default.Table_be93d8cabe2f4a27b5a47dddbb37b42e [id=551ac37765fd497283f4baeb7e207322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.050277 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6cba8aeb36441d8df42b91c38d6ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.052541 5410 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a077be5574cb8bc7a2347c6d3af00 (DEFAULT_TABLE table=default.Table_b0948279632f4c32a83404acd89ae917 [id=e62c912343e24cd4bddbaba00fb6ab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.052806 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a077be5574cb8bc7a2347c6d3af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.054911 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 1f748a59b4a347e288d534ef1296d72c (DEFAULT_TABLE table=default.Table_3c747008c670486b8d63c755d6ca129b [id=e1a39de1f5c4495b9d150f186583042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.055523 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 9501f5041dab4ca0afa406884f6c4aa1 (DEFAULT_TABLE table=default.Table_c3cca313bfc24efcacf2454a66c6e114 [id=345a522dce9445b58243d3d609349ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.055578 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 3954a4e0d3104161aad427eb2781d55d (DEFAULT_TABLE table=default.Table_558ec8354d9943f6ad3e2be4108aefc6 [id=21dd3a734fb64db0bc7b318d846fbf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.056061 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88421118434749abf4126b08b115e5 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.056385 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6cba8aeb36441d8df42b91c38d6ab3 (DEFAULT_TABLE table=default.Table_be93d8cabe2f4a27b5a47dddbb37b42e [id=551ac37765fd497283f4baeb7e207322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.056476 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9c1158e27465280e1043d91917e3e (DEFAULT_TABLE table=default.Table_398e8d420fb14944ac6abf845df55580 [id=baebad5ff3f74d25888e9b6299c7a382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.056627 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet aa485207d0f140c99a9386cb4002c480 with delete_type TABLET_DATA_DELETED (replaced by tablet 30ee9be0851243738b13c2231b554da8) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.058099 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 3954a4e0d3104161aad427eb2781d55d (DEFAULT_TABLE table=default.Table_558ec8354d9943f6ad3e2be4108aefc6 [id=21dd3a734fb64db0bc7b318d846fbf6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.059957 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 1f748a59b4a347e288d534ef1296d72c (DEFAULT_TABLE table=default.Table_3c747008c670486b8d63c755d6ca129b [id=e1a39de1f5c4495b9d150f186583042a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.060173 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9c1158e27465280e1043d91917e3e (DEFAULT_TABLE table=default.Table_398e8d420fb14944ac6abf845df55580 [id=baebad5ff3f74d25888e9b6299c7a382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.060326 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 9501f5041dab4ca0afa406884f6c4aa1 (DEFAULT_TABLE table=default.Table_c3cca313bfc24efcacf2454a66c6e114 [id=345a522dce9445b58243d3d609349ed9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.060530 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet aa485207d0f140c99a9386cb4002c480 with delete_type TABLET_DATA_DELETED (replaced by tablet 30ee9be0851243738b13c2231b554da8) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.060904 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 0b6cba8aeb36441d8df42b91c38d6ab3 (DEFAULT_TABLE table=default.Table_be93d8cabe2f4a27b5a47dddbb37b42e [id=551ac37765fd497283f4baeb7e207322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.061304 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 8c88421118434749abf4126b08b115e5 (DEFAULT_TABLE table=default.Table_5c3aae31efac4a47bc0d47090bd103c0 [id=5968089bd540475499115d04b830946e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.065402 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 52e9c1158e27465280e1043d91917e3e (DEFAULT_TABLE table=default.Table_398e8d420fb14944ac6abf845df55580 [id=baebad5ff3f74d25888e9b6299c7a382]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.065783 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9c1158e27465280e1043d91917e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.069398 5427 tablet_service.cc:1467] Processing CreateTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.069808 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4b1ceaf0b4f3385258f3703e4461b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.070112 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.070441 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227b63fbc6b4625bef0b37404b76a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.075757 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 2d913dc032b44a4fbab59b42efcba58c (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.075830 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 3e12f0849fa3423c9e88d3b4546948dc (DEFAULT_TABLE table=default.Table_98d20b30912d46dd9059405387dde59a [id=b3322647d19a472bb82de76b69d8be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.076102 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d913dc032b44a4fbab59b42efcba58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.076458 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 0034d42b2b4f4b76bf8b47186ff57a89 (DEFAULT_TABLE table=default.Table_c795df0998644b1f933ec20bdb5a2d84 [id=7b349d32a2bd4260b69952c1fe741604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.076730 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0034d42b2b4f4b76bf8b47186ff57a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.078253 5839 tablet_bootstrap.cc:492] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.078414 5839 ts_tablet_manager.cc:1367] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.400s user 0.017s sys 0.015s
I20240507 08:12:07.079169 5839 raft_consensus.cc:348] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.079404 5839 raft_consensus.cc:374] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.079475 5839 raft_consensus.cc:729] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.079927 5414 tablet_service.cc:1467] Processing CreateTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 (DEFAULT_TABLE table=default.Table_75b3c3cdeef54143a244f17d717d4e17 [id=417eb0a51f4e429898e1d44da1946aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.080283 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aaabc3e26140fda6f86ce0e518b8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.081957 5559 tablet_service.cc:1467] Processing CreateTablet for tablet cb5a077be5574cb8bc7a2347c6d3af00 (DEFAULT_TABLE table=default.Table_b0948279632f4c32a83404acd89ae917 [id=e62c912343e24cd4bddbaba00fb6ab88]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.083797 5412 tablet_service.cc:1467] Processing CreateTablet for tablet b7fe4513895d4f5d9ca8d114eb4a827f (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.084168 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fe4513895d4f5d9ca8d114eb4a827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.087329 5411 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf (DEFAULT_TABLE table=default.Table_b9d330db1b874cb487e83379f57a55e9 [id=fea0ae865a48441b9030ac72a994fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.087606 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8ad2a8324c75a1ef49e219b962cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.023072 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_89692e3b60f84ccbb00af08348cfd82d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.090324 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_89692e3b60f84ccbb00af08348cfd82d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.091354 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 3e12f0849fa3423c9e88d3b4546948dc (DEFAULT_TABLE table=default.Table_98d20b30912d46dd9059405387dde59a [id=b3322647d19a472bb82de76b69d8be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.091760 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.091822 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 0034d42b2b4f4b76bf8b47186ff57a89 (DEFAULT_TABLE table=default.Table_c795df0998644b1f933ec20bdb5a2d84 [id=7b349d32a2bd4260b69952c1fe741604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.092123 5551 tablet_service.cc:1467] Processing CreateTablet for tablet b7fe4513895d4f5d9ca8d114eb4a827f (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.092504 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 2d913dc032b44a4fbab59b42efcba58c (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.092133 5560 tablet_service.cc:1467] Processing CreateTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 (DEFAULT_TABLE table=default.Table_75b3c3cdeef54143a244f17d717d4e17 [id=417eb0a51f4e429898e1d44da1946aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.092767 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e12f0849fa3423c9e88d3b4546948dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.092914 5564 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf (DEFAULT_TABLE table=default.Table_b9d330db1b874cb487e83379f57a55e9 [id=fea0ae865a48441b9030ac72a994fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.093472 5550 tablet_service.cc:1467] Processing CreateTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.097330 5839 consensus_queue.cc:260] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.097833 5693 tablet_service.cc:1467] Processing CreateTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b (DEFAULT_TABLE table=default.Table_35e387a8fca2455ca6def29673fe2449 [id=16f44bbf352a40449df4e5122e981186]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.098990 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f (DEFAULT_TABLE table=default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.099126 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 3e12f0849fa3423c9e88d3b4546948dc (DEFAULT_TABLE table=default.Table_98d20b30912d46dd9059405387dde59a [id=b3322647d19a472bb82de76b69d8be84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.101939 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 0034d42b2b4f4b76bf8b47186ff57a89 (DEFAULT_TABLE table=default.Table_c795df0998644b1f933ec20bdb5a2d84 [id=7b349d32a2bd4260b69952c1fe741604]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.102617 5687 tablet_service.cc:1467] Processing CreateTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 (DEFAULT_TABLE table=default.Table_75b3c3cdeef54143a244f17d717d4e17 [id=417eb0a51f4e429898e1d44da1946aee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.102854 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 2d913dc032b44a4fbab59b42efcba58c (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.103062 5688 tablet_service.cc:1467] Processing CreateTablet for tablet b7fe4513895d4f5d9ca8d114eb4a827f (DEFAULT_TABLE table=default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.103457 5696 tablet_service.cc:1467] Processing CreateTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf (DEFAULT_TABLE table=default.Table_b9d330db1b874cb487e83379f57a55e9 [id=fea0ae865a48441b9030ac72a994fb53]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.108631 5839 ts_tablet_manager.cc:1398] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20240507 08:12:07.109185 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7053cfaea4a54a402c9d826613c05" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.109401 5718 raft_consensus.cc:2454] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:07.109822 5840 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:07.109956 5377 leader_election.cc:302] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.110370 6614 raft_consensus.cc:2781] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:07.110442 6614 raft_consensus.cc:482] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:07.110497 6614 raft_consensus.cc:3037] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.112589 6780 consensus_queue.cc:1035] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:07.117954 5840 tablet_bootstrap.cc:654] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.121742 5839 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.123241 5839 tablet_bootstrap.cc:654] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.124528 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b60ec15a83934999871425ef9a64ec72
W20240507 08:12:07.125815 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2606b9f58d64d588581e45611ba11b2 consensus state has no leader
I20240507 08:12:07.126121 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 1f43a80d38ed4788a13421b245fe6e8b from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:07.126060 6675 raft_consensus.cc:504] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.126772 6675 leader_election.cc:288] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.127079 5446 raft_consensus.cc:2454] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:07.128597 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f748a59b4a347e288d534ef1296d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.129196 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aaabc3e26140fda6f86ce0e518b8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.129735 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88421118434749abf4126b08b115e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.130244 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6cba8aeb36441d8df42b91c38d6ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.130570 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 723) took 2733 ms. Trace:
I20240507 08:12:07.130662 5841 rpcz_store.cc:276] 0507 08:12:04.396764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.397866 (+ 1102us) service_pool.cc:225] Handling call
0507 08:12:06.647022 (+2249156us) ts_tablet_manager.cc:1120] Deleting tablet c48da19c9b9e4207b267137d240adc5b
0507 08:12:06.647030 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:06.729973 (+ 82943us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:06.824665 (+ 94692us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.130543 (+305878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249031}
I20240507 08:12:07.131083 6513 ts_tablet_manager.cc:1888] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:07.131093 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 729) took 2733 ms. Trace:
I20240507 08:12:07.131182 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8ad2a8324c75a1ef49e219b962cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.131198 6513 log.cc:1196] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/dfa14901cdf14b72bea365b04ad521bd
I20240507 08:12:07.131191 5841 rpcz_store.cc:276] 0507 08:12:04.397371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.398020 (+ 649us) service_pool.cc:225] Handling call
0507 08:12:07.130959 (+2732939us) ts_tablet_manager.cc:1120] Deleting tablet 0c48c9720c1a4a8b849730f28f6e00d9
0507 08:12:07.130971 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.131074 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2732811}
I20240507 08:12:07.131536 6513 ts_tablet_manager.cc:1909] T dfa14901cdf14b72bea365b04ad521bd P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:07.131525 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 719) took 2735 ms. Trace:
I20240507 08:12:07.131608 5841 rpcz_store.cc:276] 0507 08:12:04.396358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.398168 (+ 1810us) service_pool.cc:225] Handling call
0507 08:12:07.131420 (+2733252us) ts_tablet_manager.cc:1120] Deleting tablet b5d4993c287a40cf964877b5464372a6
0507 08:12:07.131429 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.131510 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733125}
I20240507 08:12:07.131913 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 737) took 2599 ms. Trace:
I20240507 08:12:07.131979 5841 rpcz_store.cc:276] 0507 08:12:04.532283 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.537723 (+ 5440us) service_pool.cc:225] Handling call
0507 08:12:07.131812 (+2594089us) ts_tablet_manager.cc:1120] Deleting tablet 3b16a81588394d28945331e233e1f1db
0507 08:12:07.131820 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.131898 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2593923}
I20240507 08:12:07.132233 5841 tablet_replica.cc:330] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:07.132441 5841 raft_consensus.cc:2229] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.132594 5841 raft_consensus.cc:2258] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.133152 5841 ts_tablet_manager.cc:1875] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.133723 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501f5041dab4ca0afa406884f6c4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.134182 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9c1158e27465280e1043d91917e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.134418 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3954a4e0d3104161aad427eb2781d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.134599 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e12f0849fa3423c9e88d3b4546948dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.134886 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52e9c1158e27465280e1043d91917e3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135023 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227b63fbc6b4625bef0b37404b76a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135272 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abd47fb41bd14411bb44cc0f25e8cac4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135479 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fe4513895d4f5d9ca8d114eb4a827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135650 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1227b63fbc6b4625bef0b37404b76a6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135893 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d913dc032b44a4fbab59b42efcba58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.135994 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46cc8597ca5042ca8e85d99803d07206. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.136307 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4b1ceaf0b4f3385258f3703e4461b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.136324 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f748a59b4a347e288d534ef1296d72c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.136655 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9501f5041dab4ca0afa406884f6c4aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.136730 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0034d42b2b4f4b76bf8b47186ff57a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.137007 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b6cba8aeb36441d8df42b91c38d6ab3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.137344 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c88421118434749abf4126b08b115e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.137742 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a077be5574cb8bc7a2347c6d3af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.138075 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d913dc032b44a4fbab59b42efcba58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.138068 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:07.138437 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b15c8ad2a8324c75a1ef49e219b962cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.138304 5446 raft_consensus.cc:3037] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.138854 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8aaabc3e26140fda6f86ce0e518b8f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.138839 5656 leader_election.cc:302] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.139222 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7fe4513895d4f5d9ca8d114eb4a827f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.139554 6675 raft_consensus.cc:2781] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.139686 6675 raft_consensus.cc:686] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:07.139951 6675 consensus_queue.cc:237] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.139566 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e12f0849fa3423c9e88d3b4546948dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:07.151182 5656 consensus_peers.cc:485] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c48da19c9b9e4207b267137d240adc5b. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.159756 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb5a077be5574cb8bc7a2347c6d3af00. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.166121 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3954a4e0d3104161aad427eb2781d55d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.159858 5839 tablet_bootstrap.cc:492] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.170023 5839 ts_tablet_manager.cc:1367] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.001s
I20240507 08:12:07.171276 5843 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.171407 5843 ts_tablet_manager.cc:1367] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.201s user 0.031s sys 0.042s
I20240507 08:12:07.171993 5843 raft_consensus.cc:348] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.172241 5843 raft_consensus.cc:374] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.172309 5843 raft_consensus.cc:729] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.172508 5843 consensus_queue.cc:260] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.173015 5843 ts_tablet_manager.cc:1398] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:07.173389 5843 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.174537 5839 raft_consensus.cc:348] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.175668 5840 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:07.175801 5839 raft_consensus.cc:374] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.175943 5840 ts_tablet_manager.cc:1367] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.066s user 0.004s sys 0.000s
I20240507 08:12:07.176163 5839 raft_consensus.cc:729] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.176689 5840 raft_consensus.cc:348] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.176877 5840 raft_consensus.cc:374] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.176928 5840 raft_consensus.cc:729] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.177145 5840 consensus_queue.cc:260] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.177666 5840 ts_tablet_manager.cc:1398] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:07.177946 5840 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:07.177687 5839 consensus_queue.cc:260] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.178992 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0034d42b2b4f4b76bf8b47186ff57a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.180188 5839 ts_tablet_manager.cc:1398] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:12:07.180500 5839 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.181751 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fda4b1ceaf0b4f3385258f3703e4461b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.181989 5840 tablet_bootstrap.cc:654] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.183347 5843 tablet_bootstrap.cc:654] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.183529 5839 tablet_bootstrap.cc:654] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.184789 6837 tablet.cc:1722] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:07.189714 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet b60ec15a83934999871425ef9a64ec72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.190353 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c48da19c9b9e4207b267137d240adc5b (table default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]) successfully deleted
W20240507 08:12:07.190727 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0c48c9720c1a4a8b849730f28f6e00d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c48c9720c1a4a8b849730f28f6e00d9 already in progress: creating tablet
W20240507 08:12:07.190968 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b5d4993c287a40cf964877b5464372a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d4993c287a40cf964877b5464372a6 already in progress: creating tablet
W20240507 08:12:07.191251 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3b16a81588394d28945331e233e1f1db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b16a81588394d28945331e233e1f1db already in progress: creating tablet
I20240507 08:12:07.192960 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet b60ec15a83934999871425ef9a64ec72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.195997 6780 consensus_queue.cc:1035] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:07.201547 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:07.206403 5654 leader_election.cc:341] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.207540 5839 tablet_bootstrap.cc:492] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.207733 5839 ts_tablet_manager.cc:1367] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
W20240507 08:12:07.208451 5654 consensus_peers.cc:485] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.208715 5839 raft_consensus.cc:348] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.209422 5839 raft_consensus.cc:374] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.209533 5839 raft_consensus.cc:729] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.209796 5839 consensus_queue.cc:260] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.210481 5839 ts_tablet_manager.cc:1398] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240507 08:12:07.210774 5839 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.212421 5839 tablet_bootstrap.cc:654] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.212741 5840 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:07.212886 5840 ts_tablet_manager.cc:1367] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.000s
I20240507 08:12:07.213793 5840 raft_consensus.cc:348] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.214008 5840 raft_consensus.cc:374] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.214071 5840 raft_consensus.cc:729] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.214565 5843 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.214699 5843 ts_tablet_manager.cc:1367] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.041s user 0.002s sys 0.000s
I20240507 08:12:07.215309 5843 raft_consensus.cc:348] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.215523 5843 raft_consensus.cc:374] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.215584 5843 raft_consensus.cc:729] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.215914 5843 consensus_queue.cc:260] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.216727 5843 ts_tablet_manager.cc:1398] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:07.217036 5843 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.218585 5843 tablet_bootstrap.cc:654] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.214915 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet b60ec15a83934999871425ef9a64ec72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.217823 6779 consensus_queue.cc:1035] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:07.220505 5840 consensus_queue.cc:260] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.221031 5840 ts_tablet_manager.cc:1398] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:07.221303 5840 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:07.222587 5447 raft_consensus.cc:2454] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:07.225600 6614 raft_consensus.cc:504] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.226126 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.226847 5720 raft_consensus.cc:2454] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:07.229743 6614 leader_election.cc:288] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.229911 6547 ts_tablet_manager.cc:1888] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:07.230082 6547 log.cc:1196] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b22334378eec4ab3885d0f9d2d812a3e
I20240507 08:12:07.230458 6547 ts_tablet_manager.cc:1909] T b22334378eec4ab3885d0f9d2d812a3e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:07.231945 5840 tablet_bootstrap.cc:654] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.236378 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.238315 6614 raft_consensus.cc:482] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.238448 6614 raft_consensus.cc:504] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.239027 6614 leader_election.cc:288] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.239171 5656 leader_election.cc:302] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.239568 6779 raft_consensus.cc:2781] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.239682 6779 raft_consensus.cc:686] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:07.239934 6779 consensus_queue.cc:237] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.241823 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.242358 5378 leader_election.cc:341] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.248749 5843 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.248929 5843 ts_tablet_manager.cc:1367] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.003s
I20240507 08:12:07.254415 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7053cfaea4a54a402c9d826613c05" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:07.255108 5378 leader_election.cc:341] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.259876 6603 raft_consensus.cc:482] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.260056 6603 raft_consensus.cc:504] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.260764 6603 leader_election.cc:288] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:07.270069 5654 consensus_peers.cc:485] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:07.270320 5654 consensus_peers.cc:485] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.270613 5377 leader_election.cc:302] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.270931 6614 raft_consensus.cc:2781] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.271063 6614 raft_consensus.cc:686] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:07.271303 6614 consensus_queue.cc:237] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:07.274631 5517 leader_election.cc:341] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 1b19d8e28f164c588946ea5e81ec320e
I20240507 08:12:07.281304 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4de7053cfaea4a54a402c9d826613c05" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:07.281522 5720 raft_consensus.cc:3037] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.249557 5843 raft_consensus.cc:348] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.283133 5843 raft_consensus.cc:374] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.283242 5843 raft_consensus.cc:729] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.283488 5843 consensus_queue.cc:260] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.284133 5843 ts_tablet_manager.cc:1398] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.035s user 0.000s sys 0.001s
I20240507 08:12:07.284490 5843 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.285766 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:07.286320 5377 leader_election.cc:341] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.286439 5377 leader_election.cc:302] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.286825 6614 raft_consensus.cc:2726] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.286963 5843 tablet_bootstrap.cc:654] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.291291 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 716) took 2919 ms. Trace:
I20240507 08:12:07.291555 6513 rpcz_store.cc:276] 0507 08:12:04.372139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.372964 (+ 825us) service_pool.cc:225] Handling call
0507 08:12:06.942439 (+2569475us) ts_tablet_manager.cc:1120] Deleting tablet dfa14901cdf14b72bea365b04ad521bd
0507 08:12:06.942446 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.030408 (+ 87962us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.130957 (+100549us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.291266 (+160309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2569251}
I20240507 08:12:07.292775 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 717) took 2906 ms. Trace:
I20240507 08:12:07.292841 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet dfa14901cdf14b72bea365b04ad521bd (table default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]) successfully deleted
I20240507 08:12:07.295811 6513 rpcz_store.cc:276] 0507 08:12:04.385843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.387097 (+ 1254us) service_pool.cc:225] Handling call
0507 08:12:07.292623 (+2905526us) ts_tablet_manager.cc:1120] Deleting tablet b5d4993c287a40cf964877b5464372a6
0507 08:12:07.292634 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.292757 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2905291}
I20240507 08:12:07.309608 6835 tablet.cc:1722] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:07.310720 5843 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.310910 5843 ts_tablet_manager.cc:1367] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.005s
I20240507 08:12:07.311846 5843 raft_consensus.cc:348] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.312052 5843 raft_consensus.cc:374] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.312110 5843 raft_consensus.cc:729] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.312361 5843 consensus_queue.cc:260] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.313000 5843 ts_tablet_manager.cc:1398] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:07.313295 5843 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.315025 5843 tablet_bootstrap.cc:654] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.315577 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_96bb14500b6740f7bc08506e19f12519" } new_table_name: "default.Table_1d43b2cac0dd409ca65863a8b0c64158" modify_external_catalogs: true
I20240507 08:12:07.317016 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70cacbd76a194d649516d56327e82ae2" } modify_external_catalogs: true
I20240507 08:12:07.317174 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70cacbd76a194d649516d56327e82ae2" } modify_external_catalogs: true
I20240507 08:12:07.318399 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f7d324ad4a784569bdf917a46203715f" } modify_external_catalogs: true
I20240507 08:12:07.318555 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f7d324ad4a784569bdf917a46203715f" } modify_external_catalogs: true
I20240507 08:12:07.321323 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dd9261ef80574699827814799f609a4d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.321861 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd9261ef80574699827814799f609a4d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.324507 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 574a47894d274360891253534bf215a5 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:07.327528 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:07.327715 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_63c3ddcab80f42fdb70f3161d8639842 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 1 got 0
I20240507 08:12:07.330145 6603 raft_consensus.cc:482] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.330324 6603 raft_consensus.cc:504] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.331125 6603 leader_election.cc:288] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.332895 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:07.333379 5517 leader_election.cc:341] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:12:07.329515 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d10bbd0887e84b198a68db63e0fbd934"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.333873 5446 raft_consensus.cc:2454] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:07.334069 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d10bbd0887e84b198a68db63e0fbd934 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:07.342142 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a96a6023c78f4d1196b3d2fcf2f22329 consensus state has no leader
I20240507 08:12:07.343860 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet b2de528d9fc046f183f70388cffab6c8 (table default.Table_8cc125a833524bb3b0fef55c817ffe0e [id=317e03893acd43ef95533e458ed4346a]). Expected version 2 got 0
I20240507 08:12:07.349541 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
W20240507 08:12:07.353308 6604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240507 08:12:07.353742 6604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240507 08:12:07.353895 6604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240507 08:12:07.358657 5843 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.358872 5843 ts_tablet_manager.cc:1367] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20240507 08:12:07.359593 5843 raft_consensus.cc:348] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.359787 5843 raft_consensus.cc:374] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.359836 5843 raft_consensus.cc:729] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.360059 5843 consensus_queue.cc:260] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.360579 5843 ts_tablet_manager.cc:1398] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:07.363528 5843 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.365120 5843 tablet_bootstrap.cc:654] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:07.366374 6604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240507 08:12:07.366480 6604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240507 08:12:07.366564 6604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240507 08:12:07.366637 6604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240507 08:12:07.367115 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 91ee3a6c4ac642a3921560ea13bc1341 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.367342 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b19d8e28f164c588946ea5e81ec320e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.372048 5841 ts_tablet_manager.cc:1888] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:07.372232 5841 log.cc:1196] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/070fdbf0b00b4582a184729c2f2a4b69
I20240507 08:12:07.372627 5841 ts_tablet_manager.cc:1909] T 070fdbf0b00b4582a184729c2f2a4b69 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:07.377712 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 750) took 1935 ms. Trace:
I20240507 08:12:07.377888 6547 rpcz_store.cc:276] 0507 08:12:05.442618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.442686 (+ 68us) service_pool.cc:225] Handling call
0507 08:12:07.029274 (+1586588us) ts_tablet_manager.cc:1120] Deleting tablet b22334378eec4ab3885d0f9d2d812a3e
0507 08:12:07.029284 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.127510 (+ 98226us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.229757 (+102247us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.377688 (+147931us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586339}
I20240507 08:12:07.378799 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 751) took 1904 ms. Trace:
I20240507 08:12:07.378937 6547 rpcz_store.cc:276] 0507 08:12:05.474368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.478813 (+ 4445us) service_pool.cc:225] Handling call
0507 08:12:07.378681 (+1899868us) ts_tablet_manager.cc:1120] Deleting tablet d451d03349794726a71c255a7f177436
0507 08:12:07.378691 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.378784 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899576}
I20240507 08:12:07.379398 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 752) took 1831 ms. Trace:
I20240507 08:12:07.379521 6547 rpcz_store.cc:276] 0507 08:12:05.547920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.555669 (+ 7749us) service_pool.cc:225] Handling call
0507 08:12:07.379302 (+1823633us) ts_tablet_manager.cc:1120] Deleting tablet 2b13fbddcab5460db69f43e63f490fba
0507 08:12:07.379310 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.379387 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1823410}
I20240507 08:12:07.379952 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 753) took 1688 ms. Trace:
I20240507 08:12:07.380069 6547 rpcz_store.cc:276] 0507 08:12:05.691287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.691569 (+ 282us) service_pool.cc:225] Handling call
0507 08:12:07.379858 (+1688289us) ts_tablet_manager.cc:1120] Deleting tablet c9e2deaadfe34c2dbdc9819729cb4890
0507 08:12:07.379865 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.379940 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1688104}
I20240507 08:12:07.380231 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c89dfe31af12466bac028b49d182b934"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.380849 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c89dfe31af12466bac028b49d182b934 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.381958 6604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.246.212:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.4.246.254:37179,127.4.246.253:43459,127.4.246.252:45297
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.246.253:43459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.246.253
--webserver_port=40743
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 259fcc2b501077402e3d0e0d474a7ca475b70b5f
build type FASTDEBUG
built by None at 07 May 2024 07:43:19 UTC on efd1337a2aa3
build id 175
ASAN enabled
W20240507 08:12:07.382102 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13fa1a807d3548d6836652379d455b0d consensus state has no leader
I20240507 08:12:07.382961 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0a931f4f08a547d4b928d1ccfab56203"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.383711 6604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
W20240507 08:12:07.384060 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a931f4f08a547d4b928d1ccfab56203 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:07.385185 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89f80e643c59420a90d2d57ca0b703c7 consensus state has no leader
I20240507 08:12:07.385731 5255 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet b05fc9ed4b6549e1b8124e4e5ea72a55
I20240507 08:12:07.386063 6604 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240507 08:12:07.386001 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4a3987634ddc4464a1525c81bdb9d050" } new_table_name: "default.Table_3f83db914b5b4f38a335b48699966012" modify_external_catalogs: true
I20240507 08:12:07.387274 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5f0b399a2d074f4ca3626ff17bd5518e
I20240507 08:12:07.387974 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6f373518a45b48f8872daf99361fe15b" } modify_external_catalogs: true
I20240507 08:12:07.388140 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6f373518a45b48f8872daf99361fe15b" } modify_external_catalogs: true
I20240507 08:12:07.389375 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 68f1eeccee8e49b3a1dbd1930c2d108a from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:07.389828 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 43225934cef04109b3dbc057b230f032 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:07.390233 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed" } modify_external_catalogs: true
I20240507 08:12:07.390405 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8acb0e91e0bb4780bd95eb3f7d3520ed" } modify_external_catalogs: true
I20240507 08:12:07.391589 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 574a47894d274360891253534bf215a5
I20240507 08:12:07.392541 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_10a5e035c88f4438b00fb11dd74e9e4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.393105 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10a5e035c88f4438b00fb11dd74e9e4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.396221 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.397629 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 574a47894d274360891253534bf215a5 deleted and replaced by tablet 7eb37a5524834992aa7fc87824d242aa
I20240507 08:12:07.398975 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b78a6943f1e1466488139d09e9069803"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.399473 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78a6943f1e1466488139d09e9069803 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.400815 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8d11b4e0e2bc47629010869b9a89e8dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.401273 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d11b4e0e2bc47629010869b9a89e8dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:07.386382 5520 leader_election.cc:341] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1b19d8e28f164c588946ea5e81ec320e
I20240507 08:12:07.402370 5520 leader_election.cc:302] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:07.402779 5520 leader_election.cc:341] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:12:07.402866 5520 leader_election.cc:302] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:07.403134 6603 raft_consensus.cc:2726] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.403259 6603 raft_consensus.cc:2726] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.406484 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.396248 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a47894d274360891253534bf215a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb37a5524834992aa7fc87824d242aa) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.407681 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.409673 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.410094 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e43fa98024c444e481ad778a98287def"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.414085 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e43fa98024c444e481ad778a98287def in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:07.336973 5654 consensus_peers.cc:485] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.348987 5840 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:07.414379 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2fdba650f464a74b88fe721d2987002" } modify_external_catalogs: true
W20240507 08:12:07.414431 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 77493.2us
I20240507 08:12:07.414503 5840 ts_tablet_manager.cc:1367] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.193s user 0.018s sys 0.017s
I20240507 08:12:07.414574 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 721) took 3006 ms. Trace:
I20240507 08:12:07.414587 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2fdba650f464a74b88fe721d2987002" } modify_external_catalogs: true
I20240507 08:12:07.414649 6513 rpcz_store.cc:276] 0507 08:12:04.407730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.412541 (+ 4811us) service_pool.cc:225] Handling call
0507 08:12:07.414447 (+3001906us) ts_tablet_manager.cc:1120] Deleting tablet 0c48c9720c1a4a8b849730f28f6e00d9
0507 08:12:07.414455 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.414561 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3001706}
I20240507 08:12:07.414992 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 720) took 3007 ms. Trace:
I20240507 08:12:07.415064 6513 rpcz_store.cc:276] 0507 08:12:04.407589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.412958 (+ 5369us) service_pool.cc:225] Handling call
0507 08:12:07.414896 (+3001938us) ts_tablet_manager.cc:1120] Deleting tablet 53bc9bf1c04e4015a152cb750184c9ce
0507 08:12:07.414905 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.414979 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3001778}
I20240507 08:12:07.415331 5840 raft_consensus.cc:348] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.415503 5840 raft_consensus.cc:374] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.415555 5840 raft_consensus.cc:729] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.415763 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a47894d274360891253534bf215a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb37a5524834992aa7fc87824d242aa) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.415732 5840 consensus_queue.cc:260] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.416221 5840 ts_tablet_manager.cc:1398] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:07.416432 5840 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:07.416929 6547 tablet_replica.cc:330] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:07.417122 6547 raft_consensus.cc:2229] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.417290 6547 raft_consensus.cc:2258] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.386937 5656 leader_election.cc:302] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.417860 6547 ts_tablet_manager.cc:1875] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.418072 6508 raft_consensus.cc:2781] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.418179 6508 raft_consensus.cc:686] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:07.418401 6508 consensus_queue.cc:237] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.418727 5840 tablet_bootstrap.cc:654] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.419689 6780 consensus_queue.cc:1035] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:07.422263 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b5d4993c287a40cf964877b5464372a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d4993c287a40cf964877b5464372a6 already in progress: creating tablet
W20240507 08:12:07.422490 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0c48c9720c1a4a8b849730f28f6e00d9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c48c9720c1a4a8b849730f28f6e00d9 already in progress: creating tablet
W20240507 08:12:07.422652 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 53bc9bf1c04e4015a152cb750184c9ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53bc9bf1c04e4015a152cb750184c9ce already in progress: creating tablet
I20240507 08:12:07.429291 5443 raft_consensus.cc:2454] T 371ba277be774d32912d1174bec68b51 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:07.429888 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b22334378eec4ab3885d0f9d2d812a3e (table default.Table_ca27a6c33fe74eff87c2e753756029f9 [id=5052f2b89ad844c985f753570d0f8c15]) successfully deleted
W20240507 08:12:07.430083 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d451d03349794726a71c255a7f177436 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d451d03349794726a71c255a7f177436 already in progress: creating tablet
W20240507 08:12:07.430274 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2b13fbddcab5460db69f43e63f490fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b13fbddcab5460db69f43e63f490fba already in progress: creating tablet
W20240507 08:12:07.430601 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c9e2deaadfe34c2dbdc9819729cb4890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e2deaadfe34c2dbdc9819729cb4890 already in progress: creating tablet
I20240507 08:12:07.431145 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:07.436081 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648113d9b8164cc59ba238ace8f65b39 consensus state has no leader
I20240507 08:12:07.436460 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3f826c69443f463ca9524bf8917815ed" } new_table_name: "default.Table_bdafa405a484456fa07e870095e4daa8" modify_external_catalogs: true
I20240507 08:12:07.439024 5261 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet dae693ea8d1843668bfc2065db8ae6eb
I20240507 08:12:07.441560 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet dfd9624f4337437bb11b247048e28c08
I20240507 08:12:07.445613 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd9624f4337437bb11b247048e28c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.450402 5720 raft_consensus.cc:2454] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:07.452167 5377 leader_election.cc:302] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.452636 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd9624f4337437bb11b247048e28c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.452701 6816 raft_consensus.cc:2781] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.452834 6816 raft_consensus.cc:686] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:07.453039 6816 consensus_queue.cc:237] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.454428 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.462590 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 745) took 2852 ms. Trace:
I20240507 08:12:07.464694 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 574a47894d274360891253534bf215a5 with delete_type TABLET_DATA_DELETED (replaced by tablet 7eb37a5524834992aa7fc87824d242aa) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.464733 5841 rpcz_store.cc:276] 0507 08:12:04.609756 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.609808 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:07.132189 (+2522381us) ts_tablet_manager.cc:1120] Deleting tablet 070fdbf0b00b4582a184729c2f2a4b69
0507 08:12:07.132198 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.227174 (+ 94976us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.371928 (+144754us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.462564 (+ 90636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2522113}
I20240507 08:12:07.465627 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet dfd9624f4337437bb11b247048e28c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.467833 5841 tablet_replica.cc:330] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:07.468282 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.469038 5841 raft_consensus.cc:2229] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.469985 5841 pending_rounds.cc:62] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:07.471681 5841 pending_rounds.cc:68] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924778975399936 op_type: NO_OP noop_request { }
I20240507 08:12:07.415345 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 728) took 2884 ms. Trace:
I20240507 08:12:07.472806 5841 raft_consensus.cc:2866] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:07.472990 6513 rpcz_store.cc:276] 0507 08:12:04.531232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.535696 (+ 4464us) service_pool.cc:225] Handling call
0507 08:12:07.415255 (+2879559us) ts_tablet_manager.cc:1120] Deleting tablet 3b16a81588394d28945331e233e1f1db
0507 08:12:07.415263 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.415332 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2879338}
I20240507 08:12:07.473565 5841 raft_consensus.cc:2258] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.476686 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 070fdbf0b00b4582a184729c2f2a4b69 (table default.Table_b570163e3d2740c1abb9a44675630cfc [id=4851deb2bcf2430cb22de846f4410896]) successfully deleted
I20240507 08:12:07.476832 6513 tablet_replica.cc:330] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:07.476938 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82f4891ee00b401cbbedea716a758738 has no consensus state
I20240507 08:12:07.477005 6513 raft_consensus.cc:2229] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:07.477298 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3b16a81588394d28945331e233e1f1db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3b16a81588394d28945331e233e1f1db already in progress: creating tablet
I20240507 08:12:07.478456 6863 raft_consensus.cc:482] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.478587 6863 raft_consensus.cc:504] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.478765 5841 ts_tablet_manager.cc:1875] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.479171 6863 leader_election.cc:288] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.482918 6513 raft_consensus.cc:2258] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.483844 6614 raft_consensus.cc:482] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.483960 6614 raft_consensus.cc:504] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.484495 6614 leader_election.cc:288] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.485049 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.485508 5378 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.504000 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:07.485144 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:07.505586 5377 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.505707 5377 leader_election.cc:302] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.506067 6614 raft_consensus.cc:2726] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.506381 6513 ts_tablet_manager.cc:1875] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.509853 6779 raft_consensus.cc:482] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.510082 6779 raft_consensus.cc:504] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.510911 6779 leader_election.cc:288] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.511607 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.512269 5654 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.520443 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 38fc0180e9d34d11885b819529407930 (DEFAULT_TABLE table=default.Table_9d3bcb49239048a198510a548f153eaf [id=50b8858d42e247aa92920da31c8645a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.520752 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 38fc0180e9d34d11885b819529407930 (DEFAULT_TABLE table=default.Table_9d3bcb49239048a198510a548f153eaf [id=50b8858d42e247aa92920da31c8645a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.522272 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 38fc0180e9d34d11885b819529407930 (DEFAULT_TABLE table=default.Table_9d3bcb49239048a198510a548f153eaf [id=50b8858d42e247aa92920da31c8645a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.522439 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5aa521dd21634cd0af25652931069c41
I20240507 08:12:07.523137 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa521dd21634cd0af25652931069c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.523772 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa521dd21634cd0af25652931069c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.526751 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 516d8d52f8364666b91af7c7d4816b32 (DEFAULT_TABLE table=default.Table_77123b3ab92f4c93a46d201d8f0eb9ab [id=6080d971cb6641d7b83d8a4c0ef33e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.528952 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 516d8d52f8364666b91af7c7d4816b32 (DEFAULT_TABLE table=default.Table_77123b3ab92f4c93a46d201d8f0eb9ab [id=6080d971cb6641d7b83d8a4c0ef33e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.529407 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa521dd21634cd0af25652931069c41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.529670 6863 raft_consensus.cc:482] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.529844 6863 raft_consensus.cc:504] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.530601 6863 leader_election.cc:288] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.531140 5427 tablet_service.cc:1467] Processing CreateTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.531814 5547 tablet_service.cc:1467] Processing CreateTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.532339 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fc0180e9d34d11885b819529407930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.533015 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516d8d52f8364666b91af7c7d4816b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.533669 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6ca1a20df748e097de097d1c2d7be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.533821 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fc0180e9d34d11885b819529407930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.535539 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 (DEFAULT_TABLE table=default.Table_fdd7bcf579774a45a7fe204ceec43195 [id=819ac00611cb44e2bb3cf13a221e522b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.537268 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 (DEFAULT_TABLE table=default.Table_fdd7bcf579774a45a7fe204ceec43195 [id=819ac00611cb44e2bb3cf13a221e522b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.537895 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 (DEFAULT_TABLE table=default.Table_fdd7bcf579774a45a7fe204ceec43195 [id=819ac00611cb44e2bb3cf13a221e522b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.538288 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1a4e05273b4f8c83dc8c7825a39310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.541903 5690 tablet_service.cc:1467] Processing CreateTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.542302 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6ca1a20df748e097de097d1c2d7be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.545854 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4891ee00b401cbbedea716a758738 (DEFAULT_TABLE table=default.Table_bdafa405a484456fa07e870095e4daa8 [id=c8cf84c4109a46b0a6ba90228486e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.546209 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4891ee00b401cbbedea716a758738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.549371 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4891ee00b401cbbedea716a758738 (DEFAULT_TABLE table=default.Table_bdafa405a484456fa07e870095e4daa8 [id=c8cf84c4109a46b0a6ba90228486e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.550194 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cc196bc1f789402ab1d5fbb907534f09" } new_table_name: "default.Table_d22c1a1180864acc971b338f4ca9e231" modify_external_catalogs: true
I20240507 08:12:07.550508 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 7242c3f6a8b948949a70b61be202324f (DEFAULT_TABLE table=default.Table_ef7bea1804634eaf8efbf1bc57d940bd [id=71e7c14854764a05aaed69c7ac1bf040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.551060 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 7242c3f6a8b948949a70b61be202324f (DEFAULT_TABLE table=default.Table_ef7bea1804634eaf8efbf1bc57d940bd [id=71e7c14854764a05aaed69c7ac1bf040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.551414 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 516d8d52f8364666b91af7c7d4816b32 (DEFAULT_TABLE table=default.Table_77123b3ab92f4c93a46d201d8f0eb9ab [id=6080d971cb6641d7b83d8a4c0ef33e9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.552191 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 82f4891ee00b401cbbedea716a758738 (DEFAULT_TABLE table=default.Table_bdafa405a484456fa07e870095e4daa8 [id=c8cf84c4109a46b0a6ba90228486e423]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.552527 5546 tablet_service.cc:1467] Processing CreateTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 (DEFAULT_TABLE table=default.Table_26e6e2091f7f4938ba8bdce96b06aeb3 [id=986d26af8ccc490eb873f1b02459c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.552574 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 7242c3f6a8b948949a70b61be202324f (DEFAULT_TABLE table=default.Table_ef7bea1804634eaf8efbf1bc57d940bd [id=71e7c14854764a05aaed69c7ac1bf040]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.554344 5415 tablet_service.cc:1467] Processing CreateTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 (DEFAULT_TABLE table=default.Table_26e6e2091f7f4938ba8bdce96b06aeb3 [id=986d26af8ccc490eb873f1b02459c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.555068 5689 tablet_service.cc:1467] Processing CreateTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 (DEFAULT_TABLE table=default.Table_26e6e2091f7f4938ba8bdce96b06aeb3 [id=986d26af8ccc490eb873f1b02459c0c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.556169 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 3400e54a700040b6a3fd8a9d8f370165 (DEFAULT_TABLE table=default.Table_ef912d7f601f4332a3e635270c865ef9 [id=761a8be5d2cc49acbac74ae541dda24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.558687 5839 tablet_bootstrap.cc:492] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.558817 5839 ts_tablet_manager.cc:1367] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.348s user 0.000s sys 0.003s
I20240507 08:12:07.558799 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 3400e54a700040b6a3fd8a9d8f370165 (DEFAULT_TABLE table=default.Table_ef912d7f601f4332a3e635270c865ef9 [id=761a8be5d2cc49acbac74ae541dda24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.559501 5839 raft_consensus.cc:348] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.559679 5839 raft_consensus.cc:374] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.559751 5839 raft_consensus.cc:729] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.560015 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 3400e54a700040b6a3fd8a9d8f370165 (DEFAULT_TABLE table=default.Table_ef912d7f601f4332a3e635270c865ef9 [id=761a8be5d2cc49acbac74ae541dda24d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.560035 5839 consensus_queue.cc:260] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:07.560297 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 389aced831b547f6af5c3eff90c4d3b2 consensus state has no leader
I20240507 08:12:07.560591 5839 ts_tablet_manager.cc:1398] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 08:12:07.552569 5520 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:07.560817 5839 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.562201 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516d8d52f8364666b91af7c7d4816b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.562580 5839 tablet_bootstrap.cc:654] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.566350 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7242c3f6a8b948949a70b61be202324f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.568298 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c08c3eb77412886d06fd5321d846f (DEFAULT_TABLE table=default.Table_939d165f98ef46009d0837f15dc13dbd [id=1b291f6b47f4448ead7cb36a6369335a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.568914 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c08c3eb77412886d06fd5321d846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.569792 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1935dd9e2a34c49bb3e0021f46bc765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.573271 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4891ee00b401cbbedea716a758738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.573747 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3400e54a700040b6a3fd8a9d8f370165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.578548 5843 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.578677 5843 ts_tablet_manager.cc:1367] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.215s user 0.014s sys 0.010s
I20240507 08:12:07.579468 5843 raft_consensus.cc:348] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.579643 5843 raft_consensus.cc:374] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.579687 5843 raft_consensus.cc:729] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.579876 5843 consensus_queue.cc:260] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.580473 5843 ts_tablet_manager.cc:1398] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 08:12:07.580598 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 162373us
I20240507 08:12:07.581918 5656 leader_election.cc:302] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.582947 5840 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:07.583082 5840 ts_tablet_manager.cc:1367] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.167s user 0.010s sys 0.025s
I20240507 08:12:07.583858 5840 raft_consensus.cc:348] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.584062 5840 raft_consensus.cc:374] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.584125 5840 raft_consensus.cc:729] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.584100 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2de528d9fc046f183f70388cffab6c8" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.584328 5840 consensus_queue.cc:260] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.584801 5840 ts_tablet_manager.cc:1398] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:07.585072 5840 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:07.585655 5654 consensus_peers.cc:485] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:07.589681 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 82f4891ee00b401cbbedea716a758738 (table default.Table_bdafa405a484456fa07e870095e4daa8 [id=c8cf84c4109a46b0a6ba90228486e423], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f4891ee00b401cbbedea716a758738 consensus state has no leader
I20240507 08:12:07.592141 5839 tablet_bootstrap.cc:492] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.592268 5839 ts_tablet_manager.cc:1367] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.001s
I20240507 08:12:07.593008 5839 raft_consensus.cc:348] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.593221 5839 raft_consensus.cc:374] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.593276 5839 raft_consensus.cc:729] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.594728 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c08c3eb77412886d06fd5321d846f (DEFAULT_TABLE table=default.Table_939d165f98ef46009d0837f15dc13dbd [id=1b291f6b47f4448ead7cb36a6369335a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.596573 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 44af2d81508a4e3caf774d3643d8136b (DEFAULT_TABLE table=default.Table_91f1591433454159bdb582ad57a698d8 [id=afb75481cf704a218080f6975cf4c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.597051 5546 tablet_service.cc:1467] Processing CreateTablet for tablet a93cc557b16443ae95770626e98a6ec5 (DEFAULT_TABLE table=default.Table_83eabeb1c35e407cb08ed3a030480d7e [id=5ce872de7b63442b9f5414b73f790703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.597414 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93cc557b16443ae95770626e98a6ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.597887 5416 tablet_service.cc:1467] Processing CreateTablet for tablet a93cc557b16443ae95770626e98a6ec5 (DEFAULT_TABLE table=default.Table_83eabeb1c35e407cb08ed3a030480d7e [id=5ce872de7b63442b9f5414b73f790703]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:07.599529 5654 consensus_peers.cc:485] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:07.607509 5656 consensus_peers.cc:485] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:07.607918 5517 leader_election.cc:341] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:07.608033 5517 leader_election.cc:302] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:07.608445 5517 leader_election.cc:302] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:07.608785 6603 raft_consensus.cc:2726] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.608938 6603 raft_consensus.cc:2726] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.611603 6784 raft_consensus.cc:2781] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:07.611769 6784 raft_consensus.cc:686] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:07.612031 6784 consensus_queue.cc:237] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.613410 5840 tablet_bootstrap.cc:654] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.613947 5693 tablet_service.cc:1467] Processing CreateTablet for tablet a93cc557b16443ae95770626e98a6ec5 (DEFAULT_TABLE table=default.Table_83eabeb1c35e407cb08ed3a030480d7e [id=5ce872de7b63442b9f5414b73f790703]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.615175 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 6b8c08c3eb77412886d06fd5321d846f (DEFAULT_TABLE table=default.Table_939d165f98ef46009d0837f15dc13dbd [id=1b291f6b47f4448ead7cb36a6369335a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:07.615764 5654 consensus_peers.cc:485] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.616897 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4ae0974a73994e749cf5734401312e03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.617666 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ae0974a73994e749cf5734401312e03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.593452 5839 consensus_queue.cc:260] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.618538 5839 ts_tablet_manager.cc:1398] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240507 08:12:07.618834 5839 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.620342 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_215f0b92297748bd926f9b40fea2f149" } new_table_name: "default.Table_74303359aed849ecb2e92ecb0fe8914d" modify_external_catalogs: true
I20240507 08:12:07.624310 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bb1df10cd5d54d23bb3720be57fd9a46" } new_table_name: "default.Table_a9c53f96ea0f4ef2b792fbf2ee71a1e0" modify_external_catalogs: true
I20240507 08:12:07.628571 5839 tablet_bootstrap.cc:654] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.630645 5841 ts_tablet_manager.cc:1888] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:07.630760 5841 log.cc:1196] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:12:07.631075 5841 ts_tablet_manager.cc:1909] T b7947ca2d8324dbb9c909ec50852fed8 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:07.633960 6513 ts_tablet_manager.cc:1888] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:07.634146 6513 log.cc:1196] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4716c1285e91432a92b25ab13ada2779
I20240507 08:12:07.634568 6513 ts_tablet_manager.cc:1909] T 4716c1285e91432a92b25ab13ada2779 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:07.634941 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1935dd9e2a34c49bb3e0021f46bc765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.635594 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c08c3eb77412886d06fd5321d846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.639024 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93cc557b16443ae95770626e98a6ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.639232 5843 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:07.642109 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet aafd1dc2eaf84390b072f01a9e76232d consensus state has no leader
I20240507 08:12:07.642818 5843 tablet_bootstrap.cc:654] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:07.642849 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2572985bc2444dc7b051f8fe008e3ad4 consensus state has no leader
I20240507 08:12:07.644723 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_388837be9b1b4e38b8033dc631d6ed2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.645291 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_388837be9b1b4e38b8033dc631d6ed2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.646606 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b9fe51f8eb5e42aaa715ab338658f664" } modify_external_catalogs: true
I20240507 08:12:07.646842 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b9fe51f8eb5e42aaa715ab338658f664" } modify_external_catalogs: true
I20240507 08:12:07.643265 5839 tablet_bootstrap.cc:492] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:07.649295 5839 ts_tablet_manager.cc:1367] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.001s
I20240507 08:12:07.649782 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d67b71d4e235444db255a5638a555d43" } modify_external_catalogs: true
I20240507 08:12:07.650501 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1d55b33508cc4645b468556b1384ffda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.650924 5839 raft_consensus.cc:348] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.651113 5839 raft_consensus.cc:374] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.651083 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 44af2d81508a4e3caf774d3643d8136b (DEFAULT_TABLE table=default.Table_91f1591433454159bdb582ad57a698d8 [id=afb75481cf704a218080f6975cf4c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.651160 5839 raft_consensus.cc:729] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.651468 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44af2d81508a4e3caf774d3643d8136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.651827 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d67b71d4e235444db255a5638a555d43" } modify_external_catalogs: true
I20240507 08:12:07.652984 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_055ab044460f457787d9b3e18fea3729"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.653440 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_055ab044460f457787d9b3e18fea3729 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.653836 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet fccc127d380d4035905c1b152ab1017b
I20240507 08:12:07.648610 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3400e54a700040b6a3fd8a9d8f370165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.654544 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4659cc72cc3040e084a516672f35f2ec" } modify_external_catalogs: true
I20240507 08:12:07.654692 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4659cc72cc3040e084a516672f35f2ec" } modify_external_catalogs: true
I20240507 08:12:07.654824 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet fccc127d380d4035905c1b152ab1017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.655831 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3400e54a700040b6a3fd8a9d8f370165. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.656318 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44af2d81508a4e3caf774d3643d8136b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:07.656371 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d55b33508cc4645b468556b1384ffda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.656735 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a93cc557b16443ae95770626e98a6ec5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.657842 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b8c08c3eb77412886d06fd5321d846f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.658296 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet fccc127d380d4035905c1b152ab1017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.658788 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c76e55c0da46460a8ad96f8f966b29e7" } new_table_name: "default.Table_73451659761f47548c957eac0689a2f5" modify_external_catalogs: true
I20240507 08:12:07.660652 5839 consensus_queue.cc:260] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.661877 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5169ea9cb3ac4dad9ca73d6496c77f55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.662358 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5169ea9cb3ac4dad9ca73d6496c77f55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.663059 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 32ed504c4cf443e48ab285466b9e9b5d
I20240507 08:12:07.663128 5839 ts_tablet_manager.cc:1398] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240507 08:12:07.663733 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_84b116dd65724393b389f4455634ac10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.664057 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:07.664199 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_84b116dd65724393b389f4455634ac10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.663403 5839 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:07.665969 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_da4f3b14409245239819f1d7057f1af3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.664039 6603 raft_consensus.cc:482] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:07.666394 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da4f3b14409245239819f1d7057f1af3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.666466 6603 raft_consensus.cc:504] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.667155 6603 leader_election.cc:288] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:07.668475 5517 leader_election.cc:341] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5089db17ca904a04878d83209001c54e
I20240507 08:12:07.668792 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1a4e05273b4f8c83dc8c7825a39310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.669806 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f1b5b7cea20745ecb2e767e67c4c8b38" } modify_external_catalogs: true
I20240507 08:12:07.669875 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5bbaa170e90d41659516afd31247ef2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.670023 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f1b5b7cea20745ecb2e767e67c4c8b38" } modify_external_catalogs: true
W20240507 08:12:07.670305 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbaa170e90d41659516afd31247ef2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.671731 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7242c3f6a8b948949a70b61be202324f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.671659 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ef4a536ffde74983b9026698c5be760b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:07.672050 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef4a536ffde74983b9026698c5be760b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:07.672398 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1935dd9e2a34c49bb3e0021f46bc765. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.673276 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82f4891ee00b401cbbedea716a758738. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.674394 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bc029bc110b847f79d876ec3ab568832
I20240507 08:12:07.674852 6780 raft_consensus.cc:482] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.675015 6780 raft_consensus.cc:504] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.675364 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet bc029bc110b847f79d876ec3ab568832 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:07.675967 6780 leader_election.cc:288] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.677178 5425 tablet_service.cc:1467] Processing CreateTablet for tablet b6b9232426b948ae8634f7afd65df393 (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.679152 5417 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb228962374bda81ed55433130f401 (DEFAULT_TABLE table=default.Table_d4098fc03a7c4f77bf81551f3b649102 [id=0c48177ff1b7496f80e4f4791030e5ff]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:07.680354 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 52955.9us
I20240507 08:12:07.680420 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 38c461d7c28342459dc2abecf104f0ab (DEFAULT_TABLE table=default.Table_83aa0d0f56b448378d9c58f5d34035fa [id=70fb3502a3e04424b4f50b96f120f34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.681747 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 5855bc973d354a69a0579d9a76b3f92b (DEFAULT_TABLE table=default.Table_a69f8756b5934280bf92b31a383e0f88 [id=732163ac33ee4e67ac7435ee882937da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.682113 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5855bc973d354a69a0579d9a76b3f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.683491 5408 tablet_service.cc:1467] Processing CreateTablet for tablet c3a4127df7ec4cb083036b2abe640a30 (DEFAULT_TABLE table=default.Table_75272258956b4c93ac0a4aec66343afc [id=d93f92153a4045a6a3438254cc17c88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.683769 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a4127df7ec4cb083036b2abe640a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.684849 6603 consensus_queue.cc:1035] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:07.691903 5555 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb228962374bda81ed55433130f401 (DEFAULT_TABLE table=default.Table_d4098fc03a7c4f77bf81551f3b649102 [id=0c48177ff1b7496f80e4f4791030e5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.692078 5564 tablet_service.cc:1467] Processing CreateTablet for tablet c3a4127df7ec4cb083036b2abe640a30 (DEFAULT_TABLE table=default.Table_75272258956b4c93ac0a4aec66343afc [id=d93f92153a4045a6a3438254cc17c88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.692140 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 38c461d7c28342459dc2abecf104f0ab (DEFAULT_TABLE table=default.Table_83aa0d0f56b448378d9c58f5d34035fa [id=70fb3502a3e04424b4f50b96f120f34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.692487 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 5855bc973d354a69a0579d9a76b3f92b (DEFAULT_TABLE table=default.Table_a69f8756b5934280bf92b31a383e0f88 [id=732163ac33ee4e67ac7435ee882937da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.692890 5408 tablet_service.cc:1467] Processing CreateTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc (DEFAULT_TABLE table=default.Table_65d962c3598a4be5be40be25fc685388 [id=87f056697ab24c89981f0a064ac57275]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.693055 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c8c312b582bf4a4ab5cd690c377c47f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:07.693521 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.693836 5557 tablet_service.cc:1467] Processing CreateTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc (DEFAULT_TABLE table=default.Table_65d962c3598a4be5be40be25fc685388 [id=87f056697ab24c89981f0a064ac57275]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.694165 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.694231 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8c312b582bf4a4ab5cd690c377c47f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:07.694984 5654 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.695578 5699 tablet_service.cc:1467] Processing CreateTablet for tablet b6b9232426b948ae8634f7afd65df393 (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.696403 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet bc029bc110b847f79d876ec3ab568832 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.691318 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.696758 5700 tablet_service.cc:1514] Processing DeleteTablet for tablet 32ed504c4cf443e48ab285466b9e9b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.698623 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet fccc127d380d4035905c1b152ab1017b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:07.699258 5559 tablet_service.cc:1467] Processing CreateTablet for tablet b6b9232426b948ae8634f7afd65df393 (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.701179 5686 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb228962374bda81ed55433130f401 (DEFAULT_TABLE table=default.Table_d4098fc03a7c4f77bf81551f3b649102 [id=0c48177ff1b7496f80e4f4791030e5ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.701720 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:12:07.701905 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:07.705524 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d1a4e05273b4f8c83dc8c7825a39310. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.705920 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a4127df7ec4cb083036b2abe640a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.706081 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb228962374bda81ed55433130f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.706595 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5855bc973d354a69a0579d9a76b3f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.707602 5839 tablet_bootstrap.cc:654] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.709683 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c461d7c28342459dc2abecf104f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.712401 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:07.713909 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e7eef600626a4961ab1fae27162537bb consensus state has no leader
I20240507 08:12:07.715106 5263 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 7eb37a5524834992aa7fc87824d242aa
I20240507 08:12:07.715677 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b9232426b948ae8634f7afd65df393. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.718595 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd6ca1a20df748e097de097d1c2d7be3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.718998 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38fc0180e9d34d11885b819529407930. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.720239 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 516d8d52f8364666b91af7c7d4816b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.720880 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet bc029bc110b847f79d876ec3ab568832 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:07.721311 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7242c3f6a8b948949a70b61be202324f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.725957 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 44af2d81508a4e3caf774d3643d8136b (DEFAULT_TABLE table=default.Table_91f1591433454159bdb582ad57a698d8 [id=afb75481cf704a218080f6975cf4c98e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.726549 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44af2d81508a4e3caf774d3643d8136b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.730490 5843 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.730615 5843 ts_tablet_manager.cc:1367] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.001s
I20240507 08:12:07.731913 5843 raft_consensus.cc:348] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.732089 5843 raft_consensus.cc:374] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.732132 5843 raft_consensus.cc:729] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.732321 5843 consensus_queue.cc:260] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.733050 5843 ts_tablet_manager.cc:1398] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:07.738775 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 30ee9be0851243738b13c2231b554da8 (DEFAULT_TABLE table=default.Table_e359119a48d246408f844aca603ee8c0 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.741722 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68 (DEFAULT_TABLE table=default.Table_15c0a7e5faba41c6a569c92d6eb7d32d [id=9975ea5abd664855a5b2dec9337bb220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.742105 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.744943 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68 (DEFAULT_TABLE table=default.Table_15c0a7e5faba41c6a569c92d6eb7d32d [id=9975ea5abd664855a5b2dec9337bb220]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.744997 5420 tablet_service.cc:1467] Processing CreateTablet for tablet b271fdf0f24f4f1c975b22bcb199a017 (DEFAULT_TABLE table=default.Table_7e11a37eccea4b7ca542f722051823d5 [id=78e68de7a5124610bdc317001078c898]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.745405 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b271fdf0f24f4f1c975b22bcb199a017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.745620 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.749760 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ee9be0851243738b13c2231b554da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.753765 6816 raft_consensus.cc:482] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.753913 6816 raft_consensus.cc:504] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.754559 6816 leader_election.cc:288] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.758352 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 742) took 3149 ms. Trace:
I20240507 08:12:07.758476 5841 rpcz_store.cc:276] 0507 08:12:04.609386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.609729 (+ 343us) service_pool.cc:225] Handling call
0507 08:12:07.467776 (+2858047us) ts_tablet_manager.cc:1120] Deleting tablet b7947ca2d8324dbb9c909ec50852fed8
0507 08:12:07.467788 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.524586 (+ 56798us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.630519 (+105933us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.758330 (+127811us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":2857677}
I20240507 08:12:07.758715 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 4136485f3c644db0922aafd310784257 (DEFAULT_TABLE table=default.Table_8d06498c8a2a4d74a95609d3c3c19795 [id=d3c4e67edc1a47c987e53b97904074c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.759150 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4136485f3c644db0922aafd310784257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.761538 5409 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcc8275a7e4dc38015e1d2f9451dfc (DEFAULT_TABLE table=default.Table_38c78dde0b4f43bca6207b3c2d2dde2b [id=47fd76439b564c3d935802bfa100225a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.761947 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcc8275a7e4dc38015e1d2f9451dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.761914 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa521dd21634cd0af25652931069c41 (DEFAULT_TABLE table=default.Table_d2fdba650f464a74b88fe721d2987002 [id=95e542e0c36447d3a125c2d2d6637e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.769443 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa521dd21634cd0af25652931069c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.761992 5841 tablet_replica.cc:330] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:07.773785 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b9232426b948ae8634f7afd65df393. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.773802 5841 raft_consensus.cc:2229] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.773983 5841 raft_consensus.cc:2258] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.774313 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb228962374bda81ed55433130f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.774513 5841 ts_tablet_manager.cc:1875] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.775981 5418 tablet_service.cc:1467] Processing CreateTablet for tablet d94eb57a25bc440196abe6861ca19b95 (DEFAULT_TABLE table=default.Table_5340a70e2ea3467e844c940b4b41d9ee [id=b01ef69bc18547ad903ed9f41a62b676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.777678 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c461d7c28342459dc2abecf104f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.777860 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94eb57a25bc440196abe6861ca19b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.779533 5843 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.781543 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 65f5a93e7b95418bb61e6adfb9ca78d0 (DEFAULT_TABLE table=default.Table_3d66e54ff28e4b2799cb6391b6aba67f [id=febfcfd964074a49b3b5090d07a24e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.782387 5843 tablet_bootstrap.cc:654] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.782881 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 (DEFAULT_TABLE table=default.Table_a16c963f4cf149fd9ff050aab7c68dbc [id=4e214f9dd4234bf0bb2dcac1e366644c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.796166 5416 tablet_service.cc:1467] Processing CreateTablet for tablet b82da7a22025499b928406579ae43a50 (DEFAULT_TABLE table=default.Table_a5bc6a7273624dceb2e737e0f7df622c [id=8322dc40f7154a85a78976d81fc08ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.798964 5843 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.799229 5843 ts_tablet_manager.cc:1367] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.003s
I20240507 08:12:07.800154 5843 raft_consensus.cc:348] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.800493 5843 raft_consensus.cc:374] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.800608 5843 raft_consensus.cc:729] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.800931 5843 consensus_queue.cc:260] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.801108 6816 raft_consensus.cc:482] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.801277 6816 raft_consensus.cc:504] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.802014 6816 leader_election.cc:288] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.811105 5843 ts_tablet_manager.cc:1398] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240507 08:12:07.811448 5843 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.813062 5843 tablet_bootstrap.cc:654] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.807204 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 30ee9be0851243738b13c2231b554da8 (DEFAULT_TABLE table=default.Table_e359119a48d246408f844aca603ee8c0 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.817497 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ee9be0851243738b13c2231b554da8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:07.732347 5654 consensus_peers.cc:485] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:07.817859 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 85546.9us
I20240507 08:12:07.818892 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.819151 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:07.820464 5377 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:07.820650 5377 leader_election.cc:341] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.825240 5843 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.825465 5843 ts_tablet_manager.cc:1367] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240507 08:12:07.826130 5843 raft_consensus.cc:348] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.826370 5843 raft_consensus.cc:374] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.826542 5843 raft_consensus.cc:729] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.826766 5843 consensus_queue.cc:260] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.827378 5843 ts_tablet_manager.cc:1398] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:07.827648 5843 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.829365 5843 tablet_bootstrap.cc:654] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.842307 5843 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:07.842561 5843 ts_tablet_manager.cc:1367] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240507 08:12:07.853195 6880 raft_consensus.cc:482] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.854938 5694 tablet_service.cc:1467] Processing CreateTablet for tablet c3a4127df7ec4cb083036b2abe640a30 (DEFAULT_TABLE table=default.Table_75272258956b4c93ac0a4aec66343afc [id=d93f92153a4045a6a3438254cc17c88b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.855427 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3a4127df7ec4cb083036b2abe640a30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.855376 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 5855bc973d354a69a0579d9a76b3f92b (DEFAULT_TABLE table=default.Table_a69f8756b5934280bf92b31a383e0f88 [id=732163ac33ee4e67ac7435ee882937da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.853408 6614 raft_consensus.cc:482] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.855796 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5855bc973d354a69a0579d9a76b3f92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.855893 6614 raft_consensus.cc:504] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:07.856526 6614 leader_election.cc:288] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.853835 5843 raft_consensus.cc:348] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.856679 5843 raft_consensus.cc:374] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:07.856747 5843 raft_consensus.cc:729] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:07.856940 5843 consensus_queue.cc:260] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.857410 5843 ts_tablet_manager.cc:1398] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240507 08:12:07.854974 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 38c461d7c28342459dc2abecf104f0ab (DEFAULT_TABLE table=default.Table_83aa0d0f56b448378d9c58f5d34035fa [id=70fb3502a3e04424b4f50b96f120f34d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.857700 5843 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:07.857932 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38c461d7c28342459dc2abecf104f0ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.859019 5843 tablet_bootstrap.cc:654] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:07.859347 6878 raft_consensus.cc:482] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.859465 6878 raft_consensus.cc:504] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.860021 6878 leader_election.cc:288] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.862097 6880 raft_consensus.cc:504] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.862653 6880 leader_election.cc:288] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.869508 5687 tablet_service.cc:1467] Processing CreateTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc (DEFAULT_TABLE table=default.Table_65d962c3598a4be5be40be25fc685388 [id=87f056697ab24c89981f0a064ac57275]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.870198 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.870914 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d03349794726a71c255a7f177436" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.871256 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.871336 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:07.877117 5583 raft_consensus.cc:3037] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.877199 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d451d03349794726a71c255a7f177436" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.877736 5377 leader_election.cc:341] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:07.877911 5377 leader_election.cc:341] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:07.878052 5377 leader_election.cc:341] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.878180 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:07.878629 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 4136485f3c644db0922aafd310784257 (DEFAULT_TABLE table=default.Table_8d06498c8a2a4d74a95609d3c3c19795 [id=d3c4e67edc1a47c987e53b97904074c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.879000 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa521dd21634cd0af25652931069c41 (DEFAULT_TABLE table=default.Table_d2fdba650f464a74b88fe721d2987002 [id=95e542e0c36447d3a125c2d2d6637e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.879312 5560 tablet_service.cc:1467] Processing CreateTablet for tablet d94eb57a25bc440196abe6861ca19b95 (DEFAULT_TABLE table=default.Table_5340a70e2ea3467e844c940b4b41d9ee [id=b01ef69bc18547ad903ed9f41a62b676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.879886 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:07.880666 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:07.881791 5555 tablet_service.cc:1467] Processing CreateTablet for tablet b271fdf0f24f4f1c975b22bcb199a017 (DEFAULT_TABLE table=default.Table_7e11a37eccea4b7ca542f722051823d5 [id=78e68de7a5124610bdc317001078c898]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:07.883566 5378 leader_election.cc:341] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.883674 5378 leader_election.cc:302] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.883919 6880 raft_consensus.cc:2726] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:07.884052 5378 leader_election.cc:341] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.884132 5378 leader_election.cc:302] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:07.884451 5378 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.884526 5378 leader_election.cc:302] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.884723 6878 raft_consensus.cc:2726] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.884881 6614 raft_consensus.cc:2726] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:07.884896 5378 leader_election.cc:341] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.885399 5378 leader_election.cc:302] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.885840 6878 raft_consensus.cc:2726] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.886251 5551 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcc8275a7e4dc38015e1d2f9451dfc (DEFAULT_TABLE table=default.Table_38c78dde0b4f43bca6207b3c2d2dde2b [id=47fd76439b564c3d935802bfa100225a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.886711 6863 raft_consensus.cc:482] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.886937 6863 raft_consensus.cc:504] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.887988 6863 leader_election.cc:288] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:07.891925 5517 leader_election.cc:341] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:07.895627 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 30ee9be0851243738b13c2231b554da8 (DEFAULT_TABLE table=default.Table_e359119a48d246408f844aca603ee8c0 [id=f0c7194be3fd4cd1b0aa0626159623cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.896059 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30ee9be0851243738b13c2231b554da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.902323 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:07.902690 5443 raft_consensus.cc:2454] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:07.907938 5841 ts_tablet_manager.cc:1888] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:07.908087 5841 log.cc:1196] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/168e49ae24314a7aae2291485ddeda44
I20240507 08:12:07.891000 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 65f5a93e7b95418bb61e6adfb9ca78d0 (DEFAULT_TABLE table=default.Table_3d66e54ff28e4b2799cb6391b6aba67f [id=febfcfd964074a49b3b5090d07a24e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.914203 5584 raft_consensus.cc:3037] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:07.922041 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.923308 5378 leader_election.cc:341] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.923422 5378 leader_election.cc:302] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.925518 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 (DEFAULT_TABLE table=default.Table_a16c963f4cf149fd9ff050aab7c68dbc [id=4e214f9dd4234bf0bb2dcac1e366644c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.926929 6880 raft_consensus.cc:2726] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.934124 5841 ts_tablet_manager.cc:1909] T 168e49ae24314a7aae2291485ddeda44 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:07.935189 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 743) took 3325 ms. Trace:
I20240507 08:12:07.935288 5841 rpcz_store.cc:276] 0507 08:12:04.609497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.611964 (+ 2467us) service_pool.cc:225] Handling call
0507 08:12:07.758879 (+3146915us) ts_tablet_manager.cc:1120] Deleting tablet 168e49ae24314a7aae2291485ddeda44
0507 08:12:07.758890 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.838716 (+ 79826us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.907818 (+ 69102us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.935162 (+ 27344us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3146697}
I20240507 08:12:07.935828 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 168e49ae24314a7aae2291485ddeda44 (table default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]) successfully deleted
I20240507 08:12:07.936389 5841 tablet_replica.cc:330] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:07.936684 5841 raft_consensus.cc:2229] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:07.937031 5841 raft_consensus.cc:2258] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:07.940757 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa521dd21634cd0af25652931069c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.942113 5841 ts_tablet_manager.cc:1875] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:07.943732 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94eb57a25bc440196abe6861ca19b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.944276 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b271fdf0f24f4f1c975b22bcb199a017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.945308 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f5a93e7b95418bb61e6adfb9ca78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.945624 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c51ec8f693e4f108100c75470ead2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.946516 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcc8275a7e4dc38015e1d2f9451dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.951992 6880 raft_consensus.cc:482] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.952153 6880 raft_consensus.cc:504] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.953627 6880 leader_election.cc:288] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:07.953989 5564 tablet_service.cc:1467] Processing CreateTablet for tablet b82da7a22025499b928406579ae43a50 (DEFAULT_TABLE table=default.Table_a5bc6a7273624dceb2e737e0f7df622c [id=8322dc40f7154a85a78976d81fc08ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:07.954334 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82da7a22025499b928406579ae43a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.955183 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4136485f3c644db0922aafd310784257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:07.956238 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:07.956746 5377 leader_election.cc:341] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.957526 6880 raft_consensus.cc:482] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.957723 6880 raft_consensus.cc:504] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.960723 6880 leader_election.cc:288] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:07.960830 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.961557 5378 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.963618 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:07.964123 5378 leader_election.cc:341] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.964264 5378 leader_election.cc:302] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.964670 6880 raft_consensus.cc:2726] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.964856 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:07.966338 5377 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:07.966603 5377 leader_election.cc:302] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:07.967043 6880 raft_consensus.cc:2726] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:07.972502 6508 raft_consensus.cc:482] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.972716 6508 raft_consensus.cc:504] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.973445 6508 leader_election.cc:288] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.976904 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 734) took 3253 ms. Trace:
I20240507 08:12:07.977022 6513 rpcz_store.cc:276] 0507 08:12:04.723827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.723874 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:07.476788 (+2752914us) ts_tablet_manager.cc:1120] Deleting tablet 4716c1285e91432a92b25ab13ada2779
0507 08:12:07.476797 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.537924 (+ 61127us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.633815 (+ 95891us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:07.976838 (+343023us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2752722}
I20240507 08:12:07.977586 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 733) took 3253 ms. Trace:
I20240507 08:12:07.977689 6513 rpcz_store.cc:276] 0507 08:12:04.723677 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.724106 (+ 429us) service_pool.cc:225] Handling call
0507 08:12:07.977398 (+3253292us) ts_tablet_manager.cc:1120] Deleting tablet 279e95d9804c44eaa2c23a4ec02c84ff
0507 08:12:07.977410 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.977512 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3253137}
I20240507 08:12:07.978091 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 735) took 3218 ms. Trace:
I20240507 08:12:07.978189 6513 rpcz_store.cc:276] 0507 08:12:04.759744 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.759921 (+ 177us) service_pool.cc:225] Handling call
0507 08:12:07.977945 (+3218024us) ts_tablet_manager.cc:1120] Deleting tablet 79524df2415146ab926425c42068626c
0507 08:12:07.977955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.978038 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3217798}
I20240507 08:12:07.978551 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 736) took 3218 ms. Trace:
I20240507 08:12:07.978646 6513 rpcz_store.cc:276] 0507 08:12:04.759869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.760205 (+ 336us) service_pool.cc:225] Handling call
0507 08:12:07.978403 (+3218198us) ts_tablet_manager.cc:1120] Deleting tablet 33875331045146a9a02559a0b37d5c6f
0507 08:12:07.978412 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.978488 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3218052}
I20240507 08:12:07.979012 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 737) took 3209 ms. Trace:
I20240507 08:12:07.979094 6513 rpcz_store.cc:276] 0507 08:12:04.769721 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.769778 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:07.978866 (+3209088us) ts_tablet_manager.cc:1120] Deleting tablet 63685e9ef255477db589fa572b087b57
0507 08:12:07.978876 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.978952 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3208906}
I20240507 08:12:07.979450 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 738) took 3002 ms. Trace:
I20240507 08:12:07.979542 6513 rpcz_store.cc:276] 0507 08:12:04.977342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.977401 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:07.979314 (+3001913us) ts_tablet_manager.cc:1120] Deleting tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5
0507 08:12:07.979323 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.979399 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3001719}
I20240507 08:12:07.979913 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 739) took 2995 ms. Trace:
I20240507 08:12:07.980005 6513 rpcz_store.cc:276] 0507 08:12:04.984910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.988278 (+ 3368us) service_pool.cc:225] Handling call
0507 08:12:07.979762 (+2991484us) ts_tablet_manager.cc:1120] Deleting tablet 96ef77ef4f474107bf31395f5f30c12f
0507 08:12:07.979771 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.979844 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2991279}
I20240507 08:12:07.980366 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 740) took 2989 ms. Trace:
I20240507 08:12:07.980456 6513 rpcz_store.cc:276] 0507 08:12:04.990784 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.990941 (+ 157us) service_pool.cc:225] Handling call
0507 08:12:07.980223 (+2989282us) ts_tablet_manager.cc:1120] Deleting tablet 8a397c40278141b7bb80ea5fcdbb125b
0507 08:12:07.980233 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.980316 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2989087}
I20240507 08:12:07.980813 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 741) took 2989 ms. Trace:
I20240507 08:12:07.980895 6513 rpcz_store.cc:276] 0507 08:12:04.990893 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.990995 (+ 102us) service_pool.cc:225] Handling call
0507 08:12:07.980670 (+2989675us) ts_tablet_manager.cc:1120] Deleting tablet ed0018356e254b82995061ef7290556d
0507 08:12:07.980679 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.980762 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34,"tablet-delete.queue_time_us":2989525}
I20240507 08:12:07.981276 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 742) took 2987 ms. Trace:
I20240507 08:12:07.981367 6513 rpcz_store.cc:276] 0507 08:12:04.994125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.994173 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:07.981143 (+2986970us) ts_tablet_manager.cc:1120] Deleting tablet 4e3ea4658bb946439e4a50962444b3e2
0507 08:12:07.981153 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:07.981225 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2986812}
W20240507 08:12:07.987438 5250 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20240507 08:12:07.988655 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240507 08:12:07.993558 5378 consensus_peers.cc:485] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:07.994015 6508 raft_consensus.cc:482] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:07.994212 6508 raft_consensus.cc:504] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:07.995869 6508 leader_election.cc:288] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:07.996913 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6b9232426b948ae8634f7afd65df393. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.009121 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb228962374bda81ed55433130f401. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.013870 6513 tablet_replica.cc:330] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:08.014230 6513 raft_consensus.cc:2229] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:08.014690 6513 raft_consensus.cc:2258] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.015478 6513 ts_tablet_manager.cc:1875] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.023255 6880 raft_consensus.cc:482] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.023495 6880 raft_consensus.cc:504] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.024925 6880 leader_election.cc:288] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:08.025040 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.025774 5378 leader_election.cc:341] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.029795 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:08.030622 5377 leader_election.cc:341] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.032363 5377 leader_election.cc:302] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.032675 6878 raft_consensus.cc:2726] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.033185 6863 raft_consensus.cc:482] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.033393 6863 raft_consensus.cc:504] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.035179 5691 tablet_service.cc:1467] Processing CreateTablet for tablet b271fdf0f24f4f1c975b22bcb199a017 (DEFAULT_TABLE table=default.Table_7e11a37eccea4b7ca542f722051823d5 [id=78e68de7a5124610bdc317001078c898]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.035741 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b271fdf0f24f4f1c975b22bcb199a017. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.037223 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f5a93e7b95418bb61e6adfb9ca78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.037421 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c51ec8f693e4f108100c75470ead2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.042603 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82da7a22025499b928406579ae43a50. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:08.046666 5517 leader_election.cc:341] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:08.046912 6863 leader_election.cc:288] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.075577 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6eac6baa249eb847c5625d64f998f (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.076125 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6eac6baa249eb847c5625d64f998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.078465 6878 raft_consensus.cc:482] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.078621 6878 raft_consensus.cc:504] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:08.079384 6878 leader_election.cc:288] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:08.080237 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.080770 5378 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.084069 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:08.085964 5377 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.086156 5377 leader_election.cc:302] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.086473 6878 raft_consensus.cc:2726] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.091033 5683 tablet_service.cc:1467] Processing CreateTablet for tablet c3dcc8275a7e4dc38015e1d2f9451dfc (DEFAULT_TABLE table=default.Table_38c78dde0b4f43bca6207b3c2d2dde2b [id=47fd76439b564c3d935802bfa100225a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.091583 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3dcc8275a7e4dc38015e1d2f9451dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.092710 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:08.093043 5446 raft_consensus.cc:2454] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:08.091625 6891 tablet.cc:1722] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:08.095036 5697 tablet_service.cc:1467] Processing CreateTablet for tablet b82da7a22025499b928406579ae43a50 (DEFAULT_TABLE table=default.Table_a5bc6a7273624dceb2e737e0f7df622c [id=8322dc40f7154a85a78976d81fc08ab4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.096048 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b82da7a22025499b928406579ae43a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.101446 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 4136485f3c644db0922aafd310784257 (DEFAULT_TABLE table=default.Table_8d06498c8a2a4d74a95609d3c3c19795 [id=d3c4e67edc1a47c987e53b97904074c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.101979 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4136485f3c644db0922aafd310784257. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.105388 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:08.105659 5445 raft_consensus.cc:2454] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:08.117691 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a1808016fe6d4b5b9c3e6d42bba8d0b2" } new_table_name: "default.Table_e174ec7e722344baa23361800c5780d4" modify_external_catalogs: true
I20240507 08:12:08.118165 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa521dd21634cd0af25652931069c41 (DEFAULT_TABLE table=default.Table_d2fdba650f464a74b88fe721d2987002 [id=95e542e0c36447d3a125c2d2d6637e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.118762 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa521dd21634cd0af25652931069c41. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.120309 5685 tablet_service.cc:1467] Processing CreateTablet for tablet d94eb57a25bc440196abe6861ca19b95 (DEFAULT_TABLE table=default.Table_5340a70e2ea3467e844c940b4b41d9ee [id=b01ef69bc18547ad903ed9f41a62b676]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.120671 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d94eb57a25bc440196abe6861ca19b95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.121165 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:08.121855 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.125713 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de7cddb2c4f743f7ae02fd92061a418c consensus state has no leader
W20240507 08:12:08.128234 5654 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.128830 5654 leader_election.cc:341] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.133512 5654 consensus_peers.cc:485] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.136050 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111),
I20240507 08:12:08.136198 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:08.137914 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 65f5a93e7b95418bb61e6adfb9ca78d0 (DEFAULT_TABLE table=default.Table_3d66e54ff28e4b2799cb6391b6aba67f [id=febfcfd964074a49b3b5090d07a24e4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.138553 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65f5a93e7b95418bb61e6adfb9ca78d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.146252 6894 tablet.cc:1722] T b2de528d9fc046f183f70388cffab6c8 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:08.151510 5843 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:08.151664 5843 ts_tablet_manager.cc:1367] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.294s user 0.000s sys 0.036s
I20240507 08:12:08.152422 5843 raft_consensus.cc:348] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.152626 5843 raft_consensus.cc:374] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.152698 5843 raft_consensus.cc:729] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.152951 5843 consensus_queue.cc:260] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.153386 5843 ts_tablet_manager.cc:1398] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:08.162012 5843 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:08.163628 5843 tablet_bootstrap.cc:654] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:08.200430 6604 thread.cc:636] AWS (cloud detector) Time spent creating pthread: real 0.807s user 0.038s sys 0.083s
W20240507 08:12:08.201484 6604 thread.cc:602] AWS (cloud detector) Time spent starting thread: real 0.808s user 0.038s sys 0.083s
I20240507 08:12:08.209659 5839 tablet_bootstrap.cc:492] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:08.209820 5839 ts_tablet_manager.cc:1367] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.546s user 0.023s sys 0.052s
W20240507 08:12:08.211048 6898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240507 08:12:08.211611 6896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:12:08.211617 5839 raft_consensus.cc:348] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.211817 5839 raft_consensus.cc:374] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.211874 5839 raft_consensus.cc:729] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.212193 5839 consensus_queue.cc:260] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:08.215095 6870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240507 08:12:08.216622 6604 server_base.cc:1034] running on GCE node
I20240507 08:12:08.217753 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20240507 08:12:08.218753 5550 consensus_meta.cc:287] T 8d1a4e05273b4f8c83dc8c7825a39310 P fe673739d8084e7da5d7edd722e857d8: Time spent flushing consensus metadata: real 0.506s user 0.005s sys 0.000s
I20240507 08:12:08.221943 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:08.222198 5445 raft_consensus.cc:2454] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:08.223824 5520 leader_election.cc:341] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5089db17ca904a04878d83209001c54e
I20240507 08:12:08.223973 5520 leader_election.cc:302] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:08.225544 6886 raft_consensus.cc:2726] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.229857 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 8005100f64fd49a3ac29d34aa728fdfb from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:08.230561 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6d4810aabea24b539d53b0e43c7298ff" } modify_external_catalogs: true
I20240507 08:12:08.230825 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6d4810aabea24b539d53b0e43c7298ff" } modify_external_catalogs: true
I20240507 08:12:08.231402 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c207fdb532c4420b8289d1d9959722dd" } new_table_name: "default.Table_948f06376d4849c79cc3f222a3fcaaad" modify_external_catalogs: true
I20240507 08:12:08.231534 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_af01a22cd534413c8b74f2bdda0c70d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:08.231814 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_63c3ddcab80f42fdb70f3161d8639842" } new_table_name: "default.Table_d43fc2157a1546e39dc17b5f26047574" modify_external_catalogs: true
W20240507 08:12:08.232110 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af01a22cd534413c8b74f2bdda0c70d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.232265 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b9d330db1b874cb487e83379f57a55e9" } new_table_name: "default.Table_87151ec671d24d5d8ba8c66e0f7c6f85" modify_external_catalogs: true
I20240507 08:12:08.232467 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_323beab207874b98a5785e0dd1f9efcb" } modify_external_catalogs: true
I20240507 08:12:08.232664 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_323beab207874b98a5785e0dd1f9efcb" } modify_external_catalogs: true
I20240507 08:12:08.233297 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_48a8b33cbac2439689197979206121b0" } modify_external_catalogs: true
I20240507 08:12:08.233466 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_48a8b33cbac2439689197979206121b0" } modify_external_catalogs: true
I20240507 08:12:08.233795 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5725468c71ba4ac7aeac7fee46e95814"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.234457 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5725468c71ba4ac7aeac7fee46e95814 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.250468 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8005100f64fd49a3ac29d34aa728fdfb
I20240507 08:12:08.252439 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 8005100f64fd49a3ac29d34aa728fdfb deleted and replaced by tablet 43fa551f9e8b4f0bb3a53bf2917a08c4
I20240507 08:12:08.253957 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6f159b4cb58147b884dd65d03fe08c54" } modify_external_catalogs: true
I20240507 08:12:08.254304 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6f159b4cb58147b884dd65d03fe08c54" } modify_external_catalogs: true
I20240507 08:12:08.255702 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb with delete_type TABLET_DATA_DELETED (replaced by tablet 43fa551f9e8b4f0bb3a53bf2917a08c4) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.259896 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb with delete_type TABLET_DATA_DELETED (replaced by tablet 43fa551f9e8b4f0bb3a53bf2917a08c4) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.260522 6878 raft_consensus.cc:482] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.260635 6878 raft_consensus.cc:504] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.261185 6878 leader_election.cc:288] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:08.262501 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.263187 5378 leader_election.cc:341] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.265000 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 62895377a8ac4506a2f4e10039dbdd74 consensus state has no leader
I20240507 08:12:08.265625 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a570e39a52d84d3dbcaeb9ea005d9f41
I20240507 08:12:08.265791 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240507 08:12:08.267204 6604 hybrid_clock.cc:648] HybridClock initialized: now 1715069528267134 us; error 47 us; skew 500 ppm
I20240507 08:12:08.267680 6604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240507 08:12:08.268198 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.277045 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c40ed1afee294143ab85a8c03d080b74
W20240507 08:12:08.277117 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b15c8ad2a8324c75a1ef49e219b962cf consensus state has no leader
I20240507 08:12:08.278738 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet c40ed1afee294143ab85a8c03d080b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.282244 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet c40ed1afee294143ab85a8c03d080b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.284021 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2e53530be7694370bbab56e5fdc054c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.284686 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e53530be7694370bbab56e5fdc054c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.286171 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b9b6ac9544f74a3f8ecc609196a1b255"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.286623 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9b6ac9544f74a3f8ecc609196a1b255 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.284021 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_760b66fcd51f489d8279e16c09e9955e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.289564 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_760b66fcd51f489d8279e16c09e9955e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.290935 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_13bcd30a99834ef49f750a9bb990f583"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.291635 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13bcd30a99834ef49f750a9bb990f583 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.291080 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9b7222e8f81c442098889182635a4503"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.293176 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b7222e8f81c442098889182635a4503 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:08.297127 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e17502bb4ac64f6ca9c77a3b00ab9f4a consensus state has no leader
I20240507 08:12:08.297780 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c7bb96bb9d8f4930b531a3dd30d733c8
I20240507 08:12:08.299090 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.304576 5263 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 43fa551f9e8b4f0bb3a53bf2917a08c4
I20240507 08:12:08.305202 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.308014 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.311986 6886 raft_consensus.cc:482] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.312199 6886 raft_consensus.cc:504] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.313752 6886 leader_election.cc:288] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.314118 6880 raft_consensus.cc:482] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.314319 6880 raft_consensus.cc:504] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.315541 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:08.315954 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_67e5969a8d434e50bcd09e39a824f5c8" } new_table_name: "default.Table_c2c8b03ca2bf496d8112eb02078a24c4" modify_external_catalogs: true
I20240507 08:12:08.317757 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.318384 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.320122 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_605d0c8dfbc846a69225b3abf59b892e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.320742 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_605d0c8dfbc846a69225b3abf59b892e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.322407 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e81e95a06efd420bbaec15335ed3e556"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.323052 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e81e95a06efd420bbaec15335ed3e556 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:08.324438 5378 leader_election.cc:341] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.327913 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f" } modify_external_catalogs: true
I20240507 08:12:08.328101 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f06d52a6ba6a42eab7cc8c63b5546a3f" } modify_external_catalogs: true
I20240507 08:12:08.329654 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 19b77b6ef778454e91105e1e9b60020d (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.330482 5424 tablet_service.cc:1467] Processing CreateTablet for tablet d792e2f8ef324c8f907ee92a97ca59c4 (DEFAULT_TABLE table=default.Table_ef4a536ffde74983b9026698c5be760b [id=96b20be64615443b8b4a46ed60386089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.330957 5550 tablet_service.cc:1467] Processing CreateTablet for tablet d792e2f8ef324c8f907ee92a97ca59c4 (DEFAULT_TABLE table=default.Table_ef4a536ffde74983b9026698c5be760b [id=96b20be64615443b8b4a46ed60386089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.315501 6880 leader_election.cc:288] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:08.333781 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet d537bed29a2f47058255d2530b5c0d23 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:52 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.334302 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 2612c023740644cb9564c44f7bfe8b61 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.334719 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6eac6baa249eb847c5625d64f998f (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:08.336856 5378 consensus_peers.cc:485] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.340791 6604 webserver.cc:460] Webserver started at http://127.4.246.253:40743/ using document root <none> and password file <none>
I20240507 08:12:08.342011 5541 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 146) took 1071 ms. Trace:
I20240507 08:12:08.342162 5541 rpcz_store.cc:276] 0507 08:12:07.270494 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.271043 (+ 549us) service_pool.cc:225] Handling call
0507 08:12:07.375021 (+103978us) spinlock_profiling.cc:243] Waited 98.1 ms on lock 0x61800007148c. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2c41a12e 00007f2f2c22079f 00007f2f2ec1e768 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58
0507 08:12:08.341990 (+966969us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":225588608}
I20240507 08:12:08.342298 6604 fs_manager.cc:359] Metadata directory not provided
I20240507 08:12:08.342480 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240507 08:12:08.343380 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4716c1285e91432a92b25ab13ada2779 (table default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb [id=bd83f9b19311401a8ff3fd39e8c75e8f]) successfully deleted
W20240507 08:12:08.343640 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 279e95d9804c44eaa2c23a4ec02c84ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279e95d9804c44eaa2c23a4ec02c84ff already in progress: creating tablet
W20240507 08:12:08.343843 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 79524df2415146ab926425c42068626c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 79524df2415146ab926425c42068626c already in progress: creating tablet
W20240507 08:12:08.344056 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 33875331045146a9a02559a0b37d5c6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33875331045146a9a02559a0b37d5c6f already in progress: creating tablet
W20240507 08:12:08.344280 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 63685e9ef255477db589fa572b087b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63685e9ef255477db589fa572b087b57 already in progress: creating tablet
W20240507 08:12:08.344499 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 already in progress: creating tablet
W20240507 08:12:08.344702 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 96ef77ef4f474107bf31395f5f30c12f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96ef77ef4f474107bf31395f5f30c12f already in progress: creating tablet
W20240507 08:12:08.344965 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8a397c40278141b7bb80ea5fcdbb125b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a397c40278141b7bb80ea5fcdbb125b already in progress: creating tablet
W20240507 08:12:08.345147 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ed0018356e254b82995061ef7290556d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0018356e254b82995061ef7290556d already in progress: creating tablet
W20240507 08:12:08.345332 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4e3ea4658bb946439e4a50962444b3e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3ea4658bb946439e4a50962444b3e2 already in progress: creating tablet
I20240507 08:12:08.347036 6880 raft_consensus.cc:482] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.347160 6880 raft_consensus.cc:504] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.347713 6880 leader_election.cc:288] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:08.348220 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.348703 5378 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.349977 5520 leader_election.cc:341] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:08.350100 5520 leader_election.cc:302] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:08.350533 5520 consensus_peers.cc:485] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 193ec0b463d04835b74df80229c85cde. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:08.350673 5520 leader_election.cc:341] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:08.350757 5520 leader_election.cc:302] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:08.351068 5520 leader_election.cc:341] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:12:08.351089 6863 raft_consensus.cc:2726] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.351229 6886 raft_consensus.cc:2726] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.351814 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1553128c98468fa4283502bcb9f746 (DEFAULT_TABLE table=default.Table_5169ea9cb3ac4dad9ca73d6496c77f55 [id=c8959301fc5e4383baa410e7ff39684f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.352100 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 19b77b6ef778454e91105e1e9b60020d (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.352892 5552 tablet_service.cc:1467] Processing CreateTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 (DEFAULT_TABLE table=default.Table_0a931f4f08a547d4b928d1ccfab56203 [id=1492b7db10944b05850c89cb97112d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.353323 5551 tablet_service.cc:1467] Processing CreateTablet for tablet fc0735b78e724469a50963444d6a8aa5 (DEFAULT_TABLE table=default.Table_b78a6943f1e1466488139d09e9069803 [id=400aa21acc624f57b83c6afab6a8204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.354360 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf92632da4643cfab913cd925fccef9 (DEFAULT_TABLE table=default.Table_c89dfe31af12466bac028b49d182b934 [id=06df5d7dc1504eb1a82e57a9c3200602]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.354446 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1553128c98468fa4283502bcb9f746 (DEFAULT_TABLE table=default.Table_5169ea9cb3ac4dad9ca73d6496c77f55 [id=c8959301fc5e4383baa410e7ff39684f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.354813 5558 tablet_service.cc:1467] Processing CreateTablet for tablet b92d568b1b7846b9893ad4678af18efa (DEFAULT_TABLE table=default.Table_8d11b4e0e2bc47629010869b9a89e8dc [id=864b7888a15e4ca2ada941270f079a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.355059 5415 tablet_service.cc:1467] Processing CreateTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 (DEFAULT_TABLE table=default.Table_0a931f4f08a547d4b928d1ccfab56203 [id=1492b7db10944b05850c89cb97112d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.355559 5411 tablet_service.cc:1467] Processing CreateTablet for tablet fc0735b78e724469a50963444d6a8aa5 (DEFAULT_TABLE table=default.Table_b78a6943f1e1466488139d09e9069803 [id=400aa21acc624f57b83c6afab6a8204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.356045 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf92632da4643cfab913cd925fccef9 (DEFAULT_TABLE table=default.Table_c89dfe31af12466bac028b49d182b934 [id=06df5d7dc1504eb1a82e57a9c3200602]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.357829 5839 ts_tablet_manager.cc:1398] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.148s user 0.000s sys 0.003s
I20240507 08:12:08.358438 5839 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:08.149873 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 (DEFAULT_TABLE table=default.Table_a16c963f4cf149fd9ff050aab7c68dbc [id=4e214f9dd4234bf0bb2dcac1e366644c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.360011 5839 tablet_bootstrap.cc:654] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:08.036840 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68 (DEFAULT_TABLE table=default.Table_15c0a7e5faba41c6a569c92d6eb7d32d [id=9975ea5abd664855a5b2dec9337bb220]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:08.209796 5654 consensus_peers.cc:485] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:08.361054 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 151282us
W20240507 08:12:08.361297 5654 consensus_peers.cc:485] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.362900 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 19b77b6ef778454e91105e1e9b60020d (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:08.236431 5840 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.036s sys 0.049s
W20240507 08:12:08.253918 6675 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.692s user 0.039s sys 0.055s
W20240507 08:12:08.363598 5840 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.749s user 0.036s sys 0.049s
W20240507 08:12:08.363631 6675 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.802s user 0.039s sys 0.055s
I20240507 08:12:08.363976 5426 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:08.287339 6784 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.674s user 0.003s sys 0.003s
W20240507 08:12:08.364226 6784 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.751s user 0.003s sys 0.004s
I20240507 08:12:08.364326 5841 ts_tablet_manager.cc:1888] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.364432 5841 log.cc:1196] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/87377661d9374cef8ab1a97042b7f1b9
I20240507 08:12:08.364483 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.364817 5841 ts_tablet_manager.cc:1909] T 87377661d9374cef8ab1a97042b7f1b9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:08.365712 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b0a2dddb15451a8e6230ffb52d5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.366056 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d568b1b7846b9893ad4678af18efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.366504 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 744) took 3756 ms. Trace:
I20240507 08:12:08.366580 5841 rpcz_store.cc:276] 0507 08:12:04.609610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.612206 (+ 2596us) service_pool.cc:225] Handling call
0507 08:12:07.936329 (+3324123us) ts_tablet_manager.cc:1120] Deleting tablet 87377661d9374cef8ab1a97042b7f1b9
0507 08:12:07.936344 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.077635 (+141291us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.364190 (+286555us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.366484 (+ 2294us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3323954}
I20240507 08:12:08.367031 5841 tablet_replica.cc:330] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:08.367041 5654 consensus_peers.cc:485] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:08.367188 5654 consensus_peers.cc:485] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.367220 5841 raft_consensus.cc:2229] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:08.367462 5841 pending_rounds.cc:62] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:08.367511 5841 pending_rounds.cc:68] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924776590921728 op_type: NO_OP noop_request { }
I20240507 08:12:08.367619 5841 raft_consensus.cc:2866] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:08.367700 5841 raft_consensus.cc:2258] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.368978 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d792e2f8ef324c8f907ee92a97ca59c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.369067 5841 ts_tablet_manager.cc:1875] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.369460 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1553128c98468fa4283502bcb9f746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.369763 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.369901 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0735b78e724469a50963444d6a8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.369908 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 62d6eac6baa249eb847c5625d64f998f (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.335388 5656 leader_election.cc:302] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.370308 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf92632da4643cfab913cd925fccef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.371017 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 87377661d9374cef8ab1a97042b7f1b9 (table default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]) successfully deleted
I20240507 08:12:08.371412 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b77b6ef778454e91105e1e9b60020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.371994 6784 raft_consensus.cc:2781] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:08.372087 6784 raft_consensus.cc:482] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:08.372154 6784 raft_consensus.cc:3037] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:08.375238 5546 tablet_service.cc:1467] Processing CreateTablet for tablet cae4ffeeeb3e467ab4eb8edf014389ef (DEFAULT_TABLE table=default.Table_d10bbd0887e84b198a68db63e0fbd934 [id=5347ac7b8a9048ed8b6e869e464651c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.375430 6780 consensus_queue.cc:1035] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.375895 6780 consensus_queue.cc:1035] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.379943 5840 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:08.380100 5840 ts_tablet_manager.cc:1367] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.795s user 0.039s sys 0.050s
I20240507 08:12:08.381067 5840 raft_consensus.cc:348] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.381304 5840 raft_consensus.cc:374] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.381376 5840 raft_consensus.cc:729] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.382051 5422 tablet_service.cc:1467] Processing CreateTablet for tablet b92d568b1b7846b9893ad4678af18efa (DEFAULT_TABLE table=default.Table_8d11b4e0e2bc47629010869b9a89e8dc [id=864b7888a15e4ca2ada941270f079a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.383554 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:08.383621 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:08.383931 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:08.384023 5377 leader_election.cc:341] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.384157 5377 leader_election.cc:302] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:08.384649 5377 leader_election.cc:341] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.384768 5377 leader_election.cc:302] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.384990 6878 raft_consensus.cc:2726] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:08.385104 5377 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.385131 6878 raft_consensus.cc:2726] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.385181 5377 leader_election.cc:302] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.385357 6878 raft_consensus.cc:2726] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.387377 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet e1f9a56a95074e67bb33ceee1af48a18 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.388844 6880 consensus_queue.cc:1035] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:08.401623 5654 consensus_peers.cc:485] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.402431 5656 leader_election.cc:302] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.402586 6913 raft_consensus.cc:482] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.402791 6913 raft_consensus.cc:504] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.403173 5564 tablet_service.cc:1467] Processing CreateTablet for tablet fafc295e1cc04cf6a64e594e9bad69b2 (DEFAULT_TABLE table=default.Table_dd9261ef80574699827814799f609a4d [id=662f307e8c2a486bafceb591a87ce800]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.403343 5425 tablet_service.cc:1467] Processing CreateTablet for tablet cae4ffeeeb3e467ab4eb8edf014389ef (DEFAULT_TABLE table=default.Table_d10bbd0887e84b198a68db63e0fbd934 [id=5347ac7b8a9048ed8b6e869e464651c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.403546 6913 leader_election.cc:288] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.403748 6779 raft_consensus.cc:2781] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:08.403825 6779 raft_consensus.cc:482] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:08.403883 6779 raft_consensus.cc:3037] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:08.404036 6913 consensus_queue.cc:1035] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.405064 5565 tablet_service.cc:1467] Processing CreateTablet for tablet d863e4911ea14dd9a0bc63a97e164858 (DEFAULT_TABLE table=default.Table_89692e3b60f84ccbb00af08348cfd82d [id=22b88fe9a413473b89cf8129367746d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.405318 5426 tablet_service.cc:1467] Processing CreateTablet for tablet fafc295e1cc04cf6a64e594e9bad69b2 (DEFAULT_TABLE table=default.Table_dd9261ef80574699827814799f609a4d [id=662f307e8c2a486bafceb591a87ce800]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.406031 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6eac6baa249eb847c5625d64f998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.407167 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 497ab12edc724eef8d6e7b631662a4ab (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.408561 5548 tablet_service.cc:1467] Processing CreateTablet for tablet a9d702508be34fabb4e2b101e45c8e35 (DEFAULT_TABLE table=default.Table_da4f3b14409245239819f1d7057f1af3 [id=c958b57a48fa456aaf0ffe8b7791a6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.409504 5418 tablet_service.cc:1467] Processing CreateTablet for tablet d863e4911ea14dd9a0bc63a97e164858 (DEFAULT_TABLE table=default.Table_89692e3b60f84ccbb00af08348cfd82d [id=22b88fe9a413473b89cf8129367746d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.409835 5554 tablet_service.cc:1467] Processing CreateTablet for tablet c11eae1778e64e09a1bd0dcf644dfc39 (DEFAULT_TABLE table=default.Table_055ab044460f457787d9b3e18fea3729 [id=48a164b5372146f881fd94ace23e0de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.410118 5420 tablet_service.cc:1467] Processing CreateTablet for tablet a9d702508be34fabb4e2b101e45c8e35 (DEFAULT_TABLE table=default.Table_da4f3b14409245239819f1d7057f1af3 [id=c958b57a48fa456aaf0ffe8b7791a6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.410658 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d792e2f8ef324c8f907ee92a97ca59c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.411270 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b77b6ef778454e91105e1e9b60020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.411310 5421 tablet_service.cc:1467] Processing CreateTablet for tablet c11eae1778e64e09a1bd0dcf644dfc39 (DEFAULT_TABLE table=default.Table_055ab044460f457787d9b3e18fea3729 [id=48a164b5372146f881fd94ace23e0de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.411744 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 902b50c7dc084276a1968f08e7355679 (DEFAULT_TABLE table=default.Table_10a5e035c88f4438b00fb11dd74e9e4b [id=608708c84f3d4b5bb132bb85190ee3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.411834 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 497ab12edc724eef8d6e7b631662a4ab (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.413770 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf92632da4643cfab913cd925fccef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.414197 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0735b78e724469a50963444d6a8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.414539 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 902b50c7dc084276a1968f08e7355679 (DEFAULT_TABLE table=default.Table_10a5e035c88f4438b00fb11dd74e9e4b [id=608708c84f3d4b5bb132bb85190ee3c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.414649 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b0a2dddb15451a8e6230ffb52d5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.415028 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1553128c98468fa4283502bcb9f746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.416334 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:08.416872 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4ffeeeb3e467ab4eb8edf014389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.417419 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc295e1cc04cf6a64e594e9bad69b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.417851 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d568b1b7846b9893ad4678af18efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.404794 6913 consensus_queue.cc:1035] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20240507 08:12:08.421687 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902b50c7dc084276a1968f08e7355679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.424075 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 2bb648fe98e74248b5cd65603de58a02 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:08.424316 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d702508be34fabb4e2b101e45c8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.424724 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497ab12edc724eef8d6e7b631662a4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.425139 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11eae1778e64e09a1bd0dcf644dfc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.425168 5843 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:08.425276 5843 ts_tablet_manager.cc:1367] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.263s user 0.017s sys 0.085s
I20240507 08:12:08.425662 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d863e4911ea14dd9a0bc63a97e164858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.426040 5843 raft_consensus.cc:348] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.426214 5843 raft_consensus.cc:374] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.426268 5843 raft_consensus.cc:729] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.426502 5843 consensus_queue.cc:260] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.427039 5843 ts_tablet_manager.cc:1398] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:08.427354 5843 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:08.429157 5843 tablet_bootstrap.cc:654] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:08.430531 5654 leader_election.cc:341] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.433881 6880 consensus_queue.cc:1035] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.434094 5840 consensus_queue.cc:260] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.434644 5840 ts_tablet_manager.cc:1398] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.054s user 0.002s sys 0.000s
I20240507 08:12:08.435518 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_a843afde2c19414c9b6c0581df4cb832 [id=3350995d44de43ee814fcf2ae9ca08c9]). Expected version 1 got 0
W20240507 08:12:08.439596 5656 consensus_peers.cc:485] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 070fdbf0b00b4582a184729c2f2a4b69. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.439715 6913 consensus_queue.cc:1035] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.439937 5656 leader_election.cc:302] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.440469 6913 raft_consensus.cc:2781] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:08.440555 6913 raft_consensus.cc:482] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:08.440610 6913 raft_consensus.cc:3037] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:08.440465 5656 consensus_peers.cc:485] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.440739 5656 leader_election.cc:302] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:08.441159 5656 consensus_peers.cc:485] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 168e49ae24314a7aae2291485ddeda44. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.441493 6675 raft_consensus.cc:2781] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:08.441591 6675 raft_consensus.cc:482] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:08.441655 6675 raft_consensus.cc:3037] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:08.442350 5517 leader_election.cc:341] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:12:08.442474 5517 leader_election.cc:302] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:08.443048 6886 raft_consensus.cc:2726] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.443281 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 99a243c9597348bb96f10f8481200a7d (DEFAULT_TABLE table=default.Table_388837be9b1b4e38b8033dc631d6ed2f [id=84fe8a1fe18a46ffb507cbdc039bd01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.448416 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 99a243c9597348bb96f10f8481200a7d (DEFAULT_TABLE table=default.Table_388837be9b1b4e38b8033dc631d6ed2f [id=84fe8a1fe18a46ffb507cbdc039bd01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.448731 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 5370d022aa904900a28842b887b13187 (DEFAULT_TABLE table=default.Table_4ae0974a73994e749cf5734401312e03 [id=45ee1173fc914a238a573bedd7ca11ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.448822 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a243c9597348bb96f10f8481200a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.450594 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 5370d022aa904900a28842b887b13187 (DEFAULT_TABLE table=default.Table_4ae0974a73994e749cf5734401312e03 [id=45ee1173fc914a238a573bedd7ca11ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.450855 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.450959 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370d022aa904900a28842b887b13187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.454869 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:08.455029 5443 raft_consensus.cc:2454] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:08.457552 5258 catalog_manager.cc:5484] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:08.459111 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.459757 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:08.459985 5561 tablet_service.cc:1467] Processing CreateTablet for tablet d07221497ce7468fb86c257492cef397 (DEFAULT_TABLE table=default.Table_5bbaa170e90d41659516afd31247ef2d [id=a16594053e1e457094c727fef006564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.460209 5258 catalog_manager.cc:5484] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:08.460788 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.460809 5843 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:08.461079 5843 ts_tablet_manager.cc:1367] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.000s
I20240507 08:12:08.461866 5843 raft_consensus.cc:348] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.462824 5258 catalog_manager.cc:5484] T 81a1a42a22704974b1ef6cbd17d91854 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:08.465879 5843 raft_consensus.cc:374] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.466454 5408 tablet_service.cc:1467] Processing CreateTablet for tablet d07221497ce7468fb86c257492cef397 (DEFAULT_TABLE table=default.Table_5bbaa170e90d41659516afd31247ef2d [id=a16594053e1e457094c727fef006564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.466675 5553 tablet_service.cc:1467] Processing CreateTablet for tablet a905e345412541558b30c629a2eb86c5 (DEFAULT_TABLE table=default.Table_1d55b33508cc4645b468556b1384ffda [id=469a5ccf197d4ac4a03e1ca53d61d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.466929 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07221497ce7468fb86c257492cef397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.470714 5843 raft_consensus.cc:729] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.471041 5843 consensus_queue.cc:260] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.471818 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
I20240507 08:12:08.472752 5843 ts_tablet_manager.cc:1398] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.012s user 0.001s sys 0.000s
I20240507 08:12:08.474152 5843 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:08.479372 6780 consensus_queue.cc:1035] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.480199 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb with delete_type TABLET_DATA_DELETED (replaced by tablet 43fa551f9e8b4f0bb3a53bf2917a08c4) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.480397 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet a570e39a52d84d3dbcaeb9ea005d9f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.480523 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet c40ed1afee294143ab85a8c03d080b74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.480649 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet c7bb96bb9d8f4930b531a3dd30d733c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.481014 5695 tablet_service.cc:1467] Processing CreateTablet for tablet d792e2f8ef324c8f907ee92a97ca59c4 (DEFAULT_TABLE table=default.Table_ef4a536ffde74983b9026698c5be760b [id=96b20be64615443b8b4a46ed60386089]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.482863 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 0d1553128c98468fa4283502bcb9f746 (DEFAULT_TABLE table=default.Table_5169ea9cb3ac4dad9ca73d6496c77f55 [id=c8959301fc5e4383baa410e7ff39684f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.485679 5681 tablet_service.cc:1467] Processing CreateTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 (DEFAULT_TABLE table=default.Table_0a931f4f08a547d4b928d1ccfab56203 [id=1492b7db10944b05850c89cb97112d9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.485966 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 9cf92632da4643cfab913cd925fccef9 (DEFAULT_TABLE table=default.Table_c89dfe31af12466bac028b49d182b934 [id=06df5d7dc1504eb1a82e57a9c3200602]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.486168 6547 ts_tablet_manager.cc:1888] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 08:12:08.486284 6547 log.cc:1196] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c1c6b488b9014841899597d858bda5e2
I20240507 08:12:08.486503 5692 tablet_service.cc:1467] Processing CreateTablet for tablet b92d568b1b7846b9893ad4678af18efa (DEFAULT_TABLE table=default.Table_8d11b4e0e2bc47629010869b9a89e8dc [id=864b7888a15e4ca2ada941270f079a87]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.489290 5696 tablet_service.cc:1467] Processing CreateTablet for tablet cae4ffeeeb3e467ab4eb8edf014389ef (DEFAULT_TABLE table=default.Table_d10bbd0887e84b198a68db63e0fbd934 [id=5347ac7b8a9048ed8b6e869e464651c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.491135 5840 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:08.492027 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 049483b0533642f2b46d8fa0e9b123cc from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:08.485693 5700 tablet_service.cc:1467] Processing CreateTablet for tablet fc0735b78e724469a50963444d6a8aa5 (DEFAULT_TABLE table=default.Table_b78a6943f1e1466488139d09e9069803 [id=400aa21acc624f57b83c6afab6a8204b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.501025 5840 tablet_bootstrap.cc:654] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:08.504592 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 47690.4us
I20240507 08:12:08.505013 6780 consensus_queue.cc:1035] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.505710 5656 leader_election.cc:302] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.505990 6780 raft_consensus.cc:2781] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:08.506073 6780 raft_consensus.cc:482] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:08.506157 6780 raft_consensus.cc:3037] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:08.512423 5843 tablet_bootstrap.cc:654] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:08.524088 6919 log.cc:924] Time spent T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
W20240507 08:12:08.532914 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4ac55233449844d2bd40fee0be427e13 consensus state has no leader
I20240507 08:12:08.543354 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b78a6943f1e1466488139d09e9069803" } new_table_name: "default.Table_5188c34889e14190b52990be0457b191" modify_external_catalogs: true
I20240507 08:12:08.547755 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_55f22a879f524a07ba3a6edf9b32d822" } new_table_name: "default.Table_d0ecbb17529c4707a7b3cfd97b764d1c" modify_external_catalogs: true
I20240507 08:12:08.549438 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bd4ed42b3028472ab247218a0e87e059" } modify_external_catalogs: true
I20240507 08:12:08.549667 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bd4ed42b3028472ab247218a0e87e059" } modify_external_catalogs: true
I20240507 08:12:08.550662 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_29880271f0474162818f1334e42c2a3d" } new_table_name: "default.Table_70c8dabdda144b77a238abeacaa20e6e" modify_external_catalogs: true
I20240507 08:12:08.551527 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5725468c71ba4ac7aeac7fee46e95814" } new_table_name: "default.Table_855c7a629b1c4b48a6859a9d4d812be6" modify_external_catalogs: true
I20240507 08:12:08.557485 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a13969f58dd2425f9f93a2fadbdd178a" } modify_external_catalogs: true
I20240507 08:12:08.557451 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e3aa977674844b06a78cd9c7a4e183a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:08.557772 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a13969f58dd2425f9f93a2fadbdd178a" } modify_external_catalogs: true
W20240507 08:12:08.559365 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3aa977674844b06a78cd9c7a4e183a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.561038 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_81faed45055c4c6c9d1cb2efbd2993a2" } modify_external_catalogs: true
I20240507 08:12:08.561196 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_81faed45055c4c6c9d1cb2efbd2993a2" } modify_external_catalogs: true
W20240507 08:12:08.571959 5378 consensus_peers.cc:485] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.587018 5840 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:08.587203 5840 ts_tablet_manager.cc:1367] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.001s
W20240507 08:12:08.587716 6925 log.cc:924] Time spent T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20240507 08:12:08.587958 6547 ts_tablet_manager.cc:1909] T c1c6b488b9014841899597d858bda5e2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:08.588089 6914 log.cc:924] Time spent T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.173s user 0.000s sys 0.000s
I20240507 08:12:08.598222 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4ffeeeb3e467ab4eb8edf014389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.598410 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.598753 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902b50c7dc084276a1968f08e7355679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.598909 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d863e4911ea14dd9a0bc63a97e164858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.599227 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a243c9597348bb96f10f8481200a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.599421 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11eae1778e64e09a1bd0dcf644dfc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.599591 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d702508be34fabb4e2b101e45c8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.600332 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc295e1cc04cf6a64e594e9bad69b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.600739 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497ab12edc724eef8d6e7b631662a4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.601950 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 182124e13ca749c8a2a96d38514158b1
I20240507 08:12:08.602938 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35569f4031ae4c15a8f2653100e5551b" } modify_external_catalogs: true
I20240507 08:12:08.603205 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35569f4031ae4c15a8f2653100e5551b" } modify_external_catalogs: true
I20240507 08:12:08.605682 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a905e345412541558b30c629a2eb86c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.608698 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07221497ce7468fb86c257492cef397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.611701 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_775645191cbe4f3aafd1b0ee37db5591"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.612325 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_775645191cbe4f3aafd1b0ee37db5591 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.612356 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370d022aa904900a28842b887b13187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.619621 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 182124e13ca749c8a2a96d38514158b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:08.626204 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 124ede6467314270a2d99e0e45097c52 (table default.Table_3c1d67dd81bd47a0b00488a63e68d269 [id=153882654fb24597a14188c78128e3bb], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 124ede6467314270a2d99e0e45097c52 consensus state has no leader
I20240507 08:12:08.635939 5840 raft_consensus.cc:348] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.636771 5840 raft_consensus.cc:374] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.636900 5840 raft_consensus.cc:729] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.637228 5840 consensus_queue.cc:260] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:08.638132 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 71581.2us
W20240507 08:12:08.643982 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fc0735b78e724469a50963444d6a8aa5 consensus state has no leader
I20240507 08:12:08.644580 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d0344e2cea8344ddb56b67b65fdc2c57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.645373 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0344e2cea8344ddb56b67b65fdc2c57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.649466 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d2240fd14155495a8c97a0af96090faf
I20240507 08:12:08.650300 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_93d4d0c49c4b4bf1b92f816195fe7068"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:08.647099 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8fc65070bbae4db19c5d14b6670d599c
I20240507 08:12:08.652438 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_25e1a176f4084371aab7be282c6e6f6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.653096 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25e1a176f4084371aab7be282c6e6f6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.654249 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4892af21ff1c43deb37b30758ce5ec7c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.654896 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4892af21ff1c43deb37b30758ce5ec7c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.647637 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ec3e1823f9ff4f19be2b13cdcd706f59
I20240507 08:12:08.656883 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 077ccc3946b34008bea7b80d8aee2662 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:08.657478 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 93c68ddcfa7c40c180b6b12fabe165d6 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:08.648376 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 consensus state has no leader
I20240507 08:12:08.649245 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d033de6ad84349dab6747ef863ec4355"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.658905 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d033de6ad84349dab6747ef863ec4355 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:08.650907 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d4d0c49c4b4bf1b92f816195fe7068 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:08.647375 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1932ffe7d37455c8aa6bb83ffd11498 has no consensus state
I20240507 08:12:08.666873 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet d2240fd14155495a8c97a0af96090faf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.667126 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc65070bbae4db19c5d14b6670d599c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.667256 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.669720 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5b0c793dbc884efd92b49ae7798206a2
I20240507 08:12:08.675165 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:08.682246 5843 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:08.682399 5843 ts_tablet_manager.cc:1367] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.208s user 0.001s sys 0.013s
I20240507 08:12:08.683148 5843 raft_consensus.cc:348] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.683351 5843 raft_consensus.cc:374] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.683408 5843 raft_consensus.cc:729] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.683598 5843 consensus_queue.cc:260] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.684021 6931 raft_consensus.cc:482] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.684136 6931 raft_consensus.cc:504] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.685483 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:08.685493 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:08.686223 5378 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.686483 5377 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.686574 5377 leader_election.cc:302] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:08.692107 6513 ts_tablet_manager.cc:1888] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.692245 6513 log.cc:1196] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4fcc1d7e309946d783a524b61d5f1269
I20240507 08:12:08.692287 5416 tablet_service.cc:1467] Processing CreateTablet for tablet a905e345412541558b30c629a2eb86c5 (DEFAULT_TABLE table=default.Table_1d55b33508cc4645b468556b1384ffda [id=469a5ccf197d4ac4a03e1ca53d61d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.692607 6513 ts_tablet_manager.cc:1909] T 4fcc1d7e309946d783a524b61d5f1269 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:08.692690 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a905e345412541558b30c629a2eb86c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.694770 5841 ts_tablet_manager.cc:1888] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.695874 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 754) took 3004 ms. Trace:
I20240507 08:12:08.694876 5841 log.cc:1196] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/98aa4ec7b93943e5bf8f9758b3bb71e0
I20240507 08:12:08.695960 6547 rpcz_store.cc:276] 0507 08:12:05.691405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.691795 (+ 390us) service_pool.cc:225] Handling call
0507 08:12:07.416877 (+1725082us) ts_tablet_manager.cc:1120] Deleting tablet c1c6b488b9014841899597d858bda5e2
0507 08:12:07.416887 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.086903 (+670016us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.486065 (+399162us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.695851 (+209786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1724931}
I20240507 08:12:08.696295 6547 tablet_replica.cc:330] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.696353 5841 ts_tablet_manager.cc:1909] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:08.696466 6547 raft_consensus.cc:2229] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.696630 6547 raft_consensus.cc:2258] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.697634 6547 ts_tablet_manager.cc:1875] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.699733 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 182124e13ca749c8a2a96d38514158b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.700222 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet d2240fd14155495a8c97a0af96090faf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.700358 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc65070bbae4db19c5d14b6670d599c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.700449 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.700564 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.703945 6547 ts_tablet_manager.cc:1888] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:08.704047 6547 log.cc:1196] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/077ccc3946b34008bea7b80d8aee2662
I20240507 08:12:08.704381 6547 ts_tablet_manager.cc:1909] T 077ccc3946b34008bea7b80d8aee2662 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.707165 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b77b6ef778454e91105e1e9b60020d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.707818 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d792e2f8ef324c8f907ee92a97ca59c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.708278 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d1553128c98468fa4283502bcb9f746. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.708787 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1b0a2dddb15451a8e6230ffb52d5302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.709215 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cf92632da4643cfab913cd925fccef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.709661 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b92d568b1b7846b9893ad4678af18efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.710067 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cae4ffeeeb3e467ab4eb8edf014389ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.710501 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c51ec8f693e4f108100c75470ead2a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.710925 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ec4d1c1ed13473f91a7c7019e2b6e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.711342 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0735b78e724469a50963444d6a8aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.718686 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 748) took 4027 ms. Trace:
I20240507 08:12:08.718796 5841 rpcz_store.cc:276] 0507 08:12:04.691267 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.691580 (+ 313us) service_pool.cc:225] Handling call
0507 08:12:08.366985 (+3675405us) ts_tablet_manager.cc:1120] Deleting tablet 98aa4ec7b93943e5bf8f9758b3bb71e0
0507 08:12:08.366995 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.691210 (+324215us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.694683 (+ 3473us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.718652 (+ 23969us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3675223}
I20240507 08:12:08.719130 5841 tablet_replica.cc:330] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:08.719301 5841 raft_consensus.cc:2229] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.719411 5841 pending_rounds.cc:62] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:08.719441 5841 pending_rounds.cc:68] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924782387003392 op_type: NO_OP noop_request { }
I20240507 08:12:08.719535 5841 raft_consensus.cc:2866] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:08.719614 5841 raft_consensus.cc:2258] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.722802 6934 raft_consensus.cc:2726] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:08.725384 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]) successfully deleted
I20240507 08:12:08.727128 6931 leader_election.cc:288] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:08.728821 6784 raft_consensus.cc:504] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.729688 6784 leader_election.cc:288] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.730082 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:08.730631 5654 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.732868 6779 raft_consensus.cc:504] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.734833 6913 raft_consensus.cc:504] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.735339 6913 leader_election.cc:288] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.737321 6780 raft_consensus.cc:504] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.741025 5682 tablet_service.cc:1467] Processing CreateTablet for tablet fafc295e1cc04cf6a64e594e9bad69b2 (DEFAULT_TABLE table=default.Table_dd9261ef80574699827814799f609a4d [id=662f307e8c2a486bafceb591a87ce800]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.743419 5843 ts_tablet_manager.cc:1398] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.061s user 0.000s sys 0.002s
I20240507 08:12:08.743696 5843 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:08.744491 5691 tablet_service.cc:1467] Processing CreateTablet for tablet d863e4911ea14dd9a0bc63a97e164858 (DEFAULT_TABLE table=default.Table_89692e3b60f84ccbb00af08348cfd82d [id=22b88fe9a413473b89cf8129367746d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.745249 5843 tablet_bootstrap.cc:654] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:08.745851 6945 raft_consensus.cc:482] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:08.745965 6945 raft_consensus.cc:504] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.746549 6945 leader_election.cc:288] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.747632 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 497ab12edc724eef8d6e7b631662a4ab (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.748931 5841 ts_tablet_manager.cc:1875] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.749048 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:08.749415 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:08.749752 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:08.751978 6675 raft_consensus.cc:504] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.752597 6675 leader_election.cc:288] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.756682 5841 ts_tablet_manager.cc:1888] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.756791 5841 log.cc:1196] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b4504b1ed3d449619926fe80469d9790
I20240507 08:12:08.757187 5841 ts_tablet_manager.cc:1909] T b4504b1ed3d449619926fe80469d9790 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:08.757491 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 747) took 4066 ms. Trace:
I20240507 08:12:08.758145 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_de8250b1e1a9404f9a3c984b62bd309b" } new_table_name: "default.Table_c649ec5b00df4a4e851f765549a0d9f8" modify_external_catalogs: true
I20240507 08:12:08.768752 5697 tablet_service.cc:1467] Processing CreateTablet for tablet a9d702508be34fabb4e2b101e45c8e35 (DEFAULT_TABLE table=default.Table_da4f3b14409245239819f1d7057f1af3 [id=c958b57a48fa456aaf0ffe8b7791a6b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.770988 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 755) took 3079 ms. Trace:
I20240507 08:12:08.771080 6547 rpcz_store.cc:276] 0507 08:12:05.691518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.691968 (+ 450us) service_pool.cc:225] Handling call
0507 08:12:08.696253 (+3004285us) ts_tablet_manager.cc:1120] Deleting tablet 077ccc3946b34008bea7b80d8aee2662
0507 08:12:08.696261 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.701124 (+ 4863us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.703840 (+ 2716us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.770968 (+ 67128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004072}
I20240507 08:12:08.771817 5685 tablet_service.cc:1467] Processing CreateTablet for tablet c11eae1778e64e09a1bd0dcf644dfc39 (DEFAULT_TABLE table=default.Table_055ab044460f457787d9b3e18fea3729 [id=48a164b5372146f881fd94ace23e0de1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.772081 6547 tablet_replica.cc:330] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.772250 6547 raft_consensus.cc:2229] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.772383 6547 raft_consensus.cc:2258] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.772871 6547 ts_tablet_manager.cc:1875] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.774694 5694 tablet_service.cc:1467] Processing CreateTablet for tablet 902b50c7dc084276a1968f08e7355679 (DEFAULT_TABLE table=default.Table_10a5e035c88f4438b00fb11dd74e9e4b [id=608708c84f3d4b5bb132bb85190ee3c6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:08.778179 5654 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.778391 5654 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:08.778548 5654 leader_election.cc:341] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.779131 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:08.781813 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 62d6eac6baa249eb847c5625d64f998f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.782042 5840 ts_tablet_manager.cc:1398] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.195s user 0.001s sys 0.002s
W20240507 08:12:08.787103 5654 leader_election.cc:341] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.788829 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d9cadcb91f984f24ac8f220b42514007"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.789419 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9cadcb91f984f24ac8f220b42514007 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.791020 6604 fs_manager.cc:709] Time spent opening directory manager: real 0.440s user 0.053s sys 0.030s
I20240507 08:12:08.793468 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_45578b6c891e4ababea94a9331fa91b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.794090 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45578b6c891e4ababea94a9331fa91b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.796954 5840 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:08.797036 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 99a243c9597348bb96f10f8481200a7d (DEFAULT_TABLE table=default.Table_388837be9b1b4e38b8033dc631d6ed2f [id=84fe8a1fe18a46ffb507cbdc039bd01d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.797370 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99a243c9597348bb96f10f8481200a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.798328 5840 tablet_bootstrap.cc:654] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:08.799124 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 112631us
I20240507 08:12:08.801095 6779 leader_election.cc:288] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.803645 6547 ts_tablet_manager.cc:1888] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.803768 6547 log.cc:1196] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5554ebc621c24d459668f2cefc20b641
I20240507 08:12:08.804031 6547 ts_tablet_manager.cc:1909] T 5554ebc621c24d459668f2cefc20b641 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.804314 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 756) took 3102 ms. Trace:
I20240507 08:12:08.804384 6547 rpcz_store.cc:276] 0507 08:12:05.701410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.701458 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:08.772038 (+3070580us) ts_tablet_manager.cc:1120] Deleting tablet 5554ebc621c24d459668f2cefc20b641
0507 08:12:08.772047 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.799819 (+ 27772us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.803502 (+ 3683us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.804297 (+ 795us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3070371}
I20240507 08:12:08.804796 6547 tablet_replica.cc:330] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.804973 6547 raft_consensus.cc:2229] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.805106 6547 raft_consensus.cc:2258] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.805567 6547 ts_tablet_manager.cc:1875] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.807065 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c7be75b22b2d41878a28edd9a44b6456" } modify_external_catalogs: true
I20240507 08:12:08.807297 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c7be75b22b2d41878a28edd9a44b6456" } modify_external_catalogs: true
I20240507 08:12:08.815026 6547 ts_tablet_manager.cc:1888] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.815124 6547 log.cc:1196] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/34bebec088a34ebbb06d2e707d54da4c
I20240507 08:12:08.815367 6547 ts_tablet_manager.cc:1909] T 34bebec088a34ebbb06d2e707d54da4c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.815655 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 757) took 3018 ms. Trace:
I20240507 08:12:08.815740 6547 rpcz_store.cc:276] 0507 08:12:05.797625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.797679 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:08.804753 (+3007074us) ts_tablet_manager.cc:1120] Deleting tablet 34bebec088a34ebbb06d2e707d54da4c
0507 08:12:08.804762 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.808548 (+ 3786us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.814946 (+ 6398us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.815632 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3006845}
I20240507 08:12:08.816186 6547 tablet_replica.cc:330] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.816361 6547 raft_consensus.cc:2229] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.816509 6547 raft_consensus.cc:2258] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.817023 6547 ts_tablet_manager.cc:1875] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.822698 6547 ts_tablet_manager.cc:1888] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.822790 6547 log.cc:1196] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c33710c6b55744c4ba0843c228abded3
I20240507 08:12:08.823025 6547 ts_tablet_manager.cc:1909] T c33710c6b55744c4ba0843c228abded3 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.823261 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 764) took 2962 ms. Trace:
I20240507 08:12:08.823323 6547 rpcz_store.cc:276] 0507 08:12:05.861228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.861280 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:08.816136 (+2954856us) ts_tablet_manager.cc:1120] Deleting tablet c33710c6b55744c4ba0843c228abded3
0507 08:12:08.816149 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.819886 (+ 3737us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.822633 (+ 2747us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.823246 (+ 613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2954620}
I20240507 08:12:08.823815 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 765) took 2938 ms. Trace:
I20240507 08:12:08.823882 6547 rpcz_store.cc:276] 0507 08:12:05.885150 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.887491 (+ 2341us) service_pool.cc:225] Handling call
0507 08:12:08.823677 (+2936186us) ts_tablet_manager.cc:1120] Deleting tablet 3618b416194f4effb29b823709593359
0507 08:12:08.823688 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.823800 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935956}
I20240507 08:12:08.824331 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 787) took 2729 ms. Trace:
I20240507 08:12:08.824401 6547 rpcz_store.cc:276] 0507 08:12:06.095271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.132615 (+ 37344us) service_pool.cc:225] Handling call
0507 08:12:08.824210 (+2691595us) ts_tablet_manager.cc:1120] Deleting tablet 12e85df8b900468ba2584fd8f2e9bb2d
0507 08:12:08.824218 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.824317 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691419}
I20240507 08:12:08.824821 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 790) took 2618 ms. Trace:
I20240507 08:12:08.824889 6547 rpcz_store.cc:276] 0507 08:12:06.206131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.206186 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:08.824719 (+2618533us) ts_tablet_manager.cc:1120] Deleting tablet 9ced879f35e14106b2609c78b7f24002
0507 08:12:08.824740 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.824808 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2618276}
I20240507 08:12:08.825280 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 789) took 2619 ms. Trace:
I20240507 08:12:08.825347 6547 rpcz_store.cc:276] 0507 08:12:06.206030 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.254061 (+ 48031us) service_pool.cc:225] Handling call
0507 08:12:08.825188 (+2571127us) ts_tablet_manager.cc:1120] Deleting tablet c7d19b7cc39a4337826819a20205aecc
0507 08:12:08.825198 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.825265 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2570942}
I20240507 08:12:08.825698 6547 tablet_replica.cc:330] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.825883 6547 raft_consensus.cc:2229] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:08.826187 6547 raft_consensus.cc:2258] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.826853 6547 ts_tablet_manager.cc:1875] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.827687 5840 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:08.827837 5840 ts_tablet_manager.cc:1367] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240507 08:12:08.828528 5840 raft_consensus.cc:348] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.828711 5840 raft_consensus.cc:374] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.828768 5840 raft_consensus.cc:729] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.829003 5840 consensus_queue.cc:260] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.829659 5840 ts_tablet_manager.cc:1398] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:08.829916 5840 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:08.831399 5840 tablet_bootstrap.cc:654] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:08.832526 6547 ts_tablet_manager.cc:1888] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.832616 6547 log.cc:1196] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/193ec0b463d04835b74df80229c85cde
I20240507 08:12:08.832943 6547 ts_tablet_manager.cc:1909] T 193ec0b463d04835b74df80229c85cde P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.833202 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 788) took 2627 ms. Trace:
I20240507 08:12:08.833272 6547 rpcz_store.cc:276] 0507 08:12:06.205933 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.271623 (+ 65690us) service_pool.cc:225] Handling call
0507 08:12:08.825658 (+2554035us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:12:08.825666 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.829663 (+ 3997us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.832454 (+ 2791us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.833182 (+ 728us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553817}
I20240507 08:12:08.833779 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 791) took 2543 ms. Trace:
I20240507 08:12:08.833851 6547 rpcz_store.cc:276] 0507 08:12:06.290552 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.291714 (+ 1162us) service_pool.cc:225] Handling call
0507 08:12:08.833668 (+2541954us) ts_tablet_manager.cc:1120] Deleting tablet 6def4bf6a7554a4090a39b1823c4097c
0507 08:12:08.833677 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.833764 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2541786}
I20240507 08:12:08.834264 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 792) took 2543 ms. Trace:
I20240507 08:12:08.834344 6547 rpcz_store.cc:276] 0507 08:12:06.290680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.297474 (+ 6794us) service_pool.cc:225] Handling call
0507 08:12:08.834167 (+2536693us) ts_tablet_manager.cc:1120] Deleting tablet d33a3ccd7a5e4b9e956735443d4c484a
0507 08:12:08.834176 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.834247 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2536338}
I20240507 08:12:08.834805 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 793) took 2369 ms. Trace:
I20240507 08:12:08.834869 6547 rpcz_store.cc:276] 0507 08:12:06.465476 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.465543 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:08.834704 (+2369161us) ts_tablet_manager.cc:1120] Deleting tablet e5c1bc3385134d8fbcc94458ef0612ec
0507 08:12:08.834712 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.834789 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368905}
I20240507 08:12:08.835201 6547 tablet_replica.cc:330] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.835363 6547 raft_consensus.cc:2229] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.835482 6547 raft_consensus.cc:2258] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.835964 6547 ts_tablet_manager.cc:1875] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.835798 5841 rpcz_store.cc:276] 0507 08:12:04.691147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.691802 (+ 655us) service_pool.cc:225] Handling call
0507 08:12:08.719081 (+4027279us) ts_tablet_manager.cc:1120] Deleting tablet b4504b1ed3d449619926fe80469d9790
0507 08:12:08.719090 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.753470 (+ 34380us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.756596 (+ 3126us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.757470 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4027151}
I20240507 08:12:08.836330 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 749) took 4116 ms. Trace:
I20240507 08:12:08.836426 5841 rpcz_store.cc:276] 0507 08:12:04.720090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.721472 (+ 1382us) service_pool.cc:225] Handling call
0507 08:12:08.836214 (+4114742us) ts_tablet_manager.cc:1120] Deleting tablet 279e95d9804c44eaa2c23a4ec02c84ff
0507 08:12:08.836225 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.836314 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4114550}
I20240507 08:12:08.836685 5841 tablet_replica.cc:330] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:08.836869 5841 raft_consensus.cc:2229] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.837060 5841 raft_consensus.cc:2258] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.837608 5841 ts_tablet_manager.cc:1875] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.837633 5694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 902b50c7dc084276a1968f08e7355679. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.838287 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c11eae1778e64e09a1bd0dcf644dfc39. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.840456 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 497ab12edc724eef8d6e7b631662a4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.841203 6780 leader_election.cc:288] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:08.842505 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:08.843147 5654 leader_election.cc:341] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:08.843499 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d863e4911ea14dd9a0bc63a97e164858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.844125 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d702508be34fabb4e2b101e45c8e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.845467 6547 ts_tablet_manager.cc:1888] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.845788 6547 log.cc:1196] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/907b5cb0e7a54903abe0b0d0e8d617dc
I20240507 08:12:08.846467 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fafc295e1cc04cf6a64e594e9bad69b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.846782 6547 ts_tablet_manager.cc:1909] T 907b5cb0e7a54903abe0b0d0e8d617dc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.847491 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 794) took 2381 ms. Trace:
I20240507 08:12:08.847777 6547 rpcz_store.cc:276] 0507 08:12:06.466295 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.466349 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:08.835159 (+2368810us) ts_tablet_manager.cc:1120] Deleting tablet 907b5cb0e7a54903abe0b0d0e8d617dc
0507 08:12:08.835167 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.839633 (+ 4466us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.845395 (+ 5762us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.847455 (+ 2060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368642}
I20240507 08:12:08.848816 6547 tablet_replica.cc:330] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.849285 6547 raft_consensus.cc:2229] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.849720 6547 raft_consensus.cc:2258] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.850049 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 745) took 3843 ms. Trace:
I20240507 08:12:08.850059 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_48b51787d9ce4ff1b9b8744ed86885bb [id=0a1bc06a351549d5aea075ad8d026476]) successfully deleted
I20240507 08:12:08.850088 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8dd482935ec64031b96ef89534429ac3" } new_table_name: "default.Table_238daf40648c421cbd0954bcbce05338" modify_external_catalogs: true
I20240507 08:12:08.850133 6513 rpcz_store.cc:276] 0507 08:12:05.006180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.006316 (+ 136us) service_pool.cc:225] Handling call
0507 08:12:08.013678 (+3007362us) ts_tablet_manager.cc:1120] Deleting tablet 4fcc1d7e309946d783a524b61d5f1269
0507 08:12:08.013692 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.591607 (+577915us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.691994 (+100387us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.850028 (+158034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007160}
I20240507 08:12:08.850265 6547 ts_tablet_manager.cc:1875] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.850435 6513 tablet_replica.cc:330] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:08.850602 6513 raft_consensus.cc:2229] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:08.850903 6513 raft_consensus.cc:2258] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.851382 6513 ts_tablet_manager.cc:1875] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.851733 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]) successfully deleted
I20240507 08:12:08.852016 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5554ebc621c24d459668f2cefc20b641 (table default.Table_31c7cf5701db40cebfa0c46ebaae046c [id=05cdaba2cbb647cdaefe82488553cca0]) successfully deleted
I20240507 08:12:08.845906 5839 tablet_bootstrap.cc:492] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:08.852164 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 34bebec088a34ebbb06d2e707d54da4c (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
I20240507 08:12:08.852264 5839 ts_tablet_manager.cc:1367] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.494s user 0.019s sys 0.020s
I20240507 08:12:08.852317 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c33710c6b55744c4ba0843c228abded3 (table default.Table_12be08cdc73d4d488affa98e5f2d2653 [id=90f3f62a07ae4a4992abd7947e78f0be]) successfully deleted
W20240507 08:12:08.852483 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3618b416194f4effb29b823709593359 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3618b416194f4effb29b823709593359 already in progress: creating tablet
W20240507 08:12:08.853243 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 12e85df8b900468ba2584fd8f2e9bb2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e85df8b900468ba2584fd8f2e9bb2d already in progress: creating tablet
W20240507 08:12:08.853457 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9ced879f35e14106b2609c78b7f24002 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ced879f35e14106b2609c78b7f24002 already in progress: creating tablet
I20240507 08:12:08.853849 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c4186447e2f943c5b1eedcb89625e756"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.854542 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4186447e2f943c5b1eedcb89625e756 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.857944 6547 ts_tablet_manager.cc:1888] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.858081 6547 log.cc:1196] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/973a199a533b4bbea3c2615acdbf95ca
I20240507 08:12:08.858383 6547 ts_tablet_manager.cc:1909] T 973a199a533b4bbea3c2615acdbf95ca P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.858777 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 795) took 2392 ms. Trace:
I20240507 08:12:08.858891 6547 rpcz_store.cc:276] 0507 08:12:06.466781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.466835 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:08.848705 (+2381870us) ts_tablet_manager.cc:1120] Deleting tablet 973a199a533b4bbea3c2615acdbf95ca
0507 08:12:08.848724 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.853889 (+ 5165us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.857878 (+ 3989us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.858761 (+ 883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381700}
I20240507 08:12:08.859405 6547 tablet_replica.cc:330] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.859303 5839 raft_consensus.cc:348] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:08.859627 6547 raft_consensus.cc:2229] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.860947 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b4504b1ed3d449619926fe80469d9790 (table default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]) successfully deleted
I20240507 08:12:08.860981 5839 raft_consensus.cc:374] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.861054 5839 raft_consensus.cc:729] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.861194 6547 raft_consensus.cc:2258] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:08.861291 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 279e95d9804c44eaa2c23a4ec02c84ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 279e95d9804c44eaa2c23a4ec02c84ff already in progress: creating tablet
I20240507 08:12:08.861886 6547 ts_tablet_manager.cc:1875] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.862028 6955 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240507 08:12:08.862458 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c7d19b7cc39a4337826819a20205aecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7d19b7cc39a4337826819a20205aecc already in progress: creating tablet
I20240507 08:12:08.862733 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 193ec0b463d04835b74df80229c85cde (table default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]) successfully deleted
W20240507 08:12:08.863026 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6def4bf6a7554a4090a39b1823c4097c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6def4bf6a7554a4090a39b1823c4097c already in progress: creating tablet
W20240507 08:12:08.863296 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d33a3ccd7a5e4b9e956735443d4c484a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d33a3ccd7a5e4b9e956735443d4c484a already in progress: creating tablet
W20240507 08:12:08.863623 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e5c1bc3385134d8fbcc94458ef0612ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c1bc3385134d8fbcc94458ef0612ec already in progress: creating tablet
I20240507 08:12:08.863859 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 907b5cb0e7a54903abe0b0d0e8d617dc (table default.Table_74a9e58e257645d6b826cab9e4134f48 [id=0e277dd692e842b28638b8675da40ec9]) successfully deleted
W20240507 08:12:08.864114 5654 consensus_peers.cc:485] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.864114 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 973a199a533b4bbea3c2615acdbf95ca (table default.Table_964368f3c7824528b36f3fb37bb2e215 [id=4ecddc64b3da4de5a70e7a211f9ea037]) successfully deleted
I20240507 08:12:08.866107 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_493ded421178438798c30ff021b4531b" } new_table_name: "default.Table_4c741b9eb00a4e27895027010bdb5ae7" modify_external_catalogs: true
I20240507 08:12:08.867094 6956 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:08.867830 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f2eabb2165474aadb0528bf86cbc1718" } modify_external_catalogs: true
I20240507 08:12:08.868043 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f2eabb2165474aadb0528bf86cbc1718" } modify_external_catalogs: true
I20240507 08:12:08.868672 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 76c06edeabdd462292ca8c23d9cba62f (DEFAULT_TABLE table=default.Table_9b7222e8f81c442098889182635a4503 [id=8c8e7c490a284e8cb2ea7f301b1443fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.869051 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c06edeabdd462292ca8c23d9cba62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.869882 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4fcc1d7e309946d783a524b61d5f1269 (table default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]) successfully deleted
I20240507 08:12:08.872735 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_73451659761f47548c957eac0689a2f5" } modify_external_catalogs: true
I20240507 08:12:08.872906 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_73451659761f47548c957eac0689a2f5" } modify_external_catalogs: true
I20240507 08:12:08.861500 5839 consensus_queue.cc:260] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:08.870246 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f932b65c022243ba8acd49cc96e5dff1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.876461 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f932b65c022243ba8acd49cc96e5dff1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.877302 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.877517 5421 tablet_service.cc:1467] Processing CreateTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 (DEFAULT_TABLE table=default.Table_855c7a629b1c4b48a6859a9d4d812be6 [id=8d1b3ce281004cebaa989dadd5dee2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.877962 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1932ffe7d37455c8aa6bb83ffd11498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.882681 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 03c2be5fbb2743daad3e6db48bd3996d (DEFAULT_TABLE table=default.Table_13bcd30a99834ef49f750a9bb990f583 [id=c346662349e84140b4b7d216a7496e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.883033 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c2be5fbb2743daad3e6db48bd3996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.884574 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:08.890102 5427 tablet_service.cc:1467] Processing CreateTablet for tablet d026ebe0217b4e4faee751e50461d0e4 (DEFAULT_TABLE table=default.Table_af01a22cd534413c8b74f2bdda0c70d0 [id=5da7241b01394a08b844b74cd2fb05a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.890133 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 (DEFAULT_TABLE table=default.Table_760b66fcd51f489d8279e16c09e9955e [id=a3fd2f8ee2c4431b8e6d08b612bfb2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.890897 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e5e67df91f4029bf55e5a82dc31f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.890851 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.891134 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad4ed7878454fea930b3982bcb6eed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.893225 6547 ts_tablet_manager.cc:1888] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:08.893970 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d026ebe0217b4e4faee751e50461d0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.899068 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 76c06edeabdd462292ca8c23d9cba62f (DEFAULT_TABLE table=default.Table_9b7222e8f81c442098889182635a4503 [id=8c8e7c490a284e8cb2ea7f301b1443fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.900993 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.901125 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c06edeabdd462292ca8c23d9cba62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.901305 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad4ed7878454fea930b3982bcb6eed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.899667 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 620214b5c5bc43c98df65932e1eed2db (DEFAULT_TABLE table=default.Table_b9b6ac9544f74a3f8ecc609196a1b255 [id=d529797cb7bc4d5fb0c2598e0b8b8e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.900206 5562 tablet_service.cc:1467] Processing CreateTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 (DEFAULT_TABLE table=default.Table_855c7a629b1c4b48a6859a9d4d812be6 [id=8d1b3ce281004cebaa989dadd5dee2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.901738 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1932ffe7d37455c8aa6bb83ffd11498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.900758 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 03c2be5fbb2743daad3e6db48bd3996d (DEFAULT_TABLE table=default.Table_13bcd30a99834ef49f750a9bb990f583 [id=c346662349e84140b4b7d216a7496e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.905382 5839 ts_tablet_manager.cc:1398] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240507 08:12:08.905421 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c2be5fbb2743daad3e6db48bd3996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.907300 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 620214b5c5bc43c98df65932e1eed2db (DEFAULT_TABLE table=default.Table_b9b6ac9544f74a3f8ecc609196a1b255 [id=d529797cb7bc4d5fb0c2598e0b8b8e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.907637 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620214b5c5bc43c98df65932e1eed2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.909339 5839 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:08.911455 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620214b5c5bc43c98df65932e1eed2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.912498 6513 ts_tablet_manager.cc:1888] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.912612 6513 log.cc:1196] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c48da19c9b9e4207b267137d240adc5b
I20240507 08:12:08.912954 6513 ts_tablet_manager.cc:1909] T c48da19c9b9e4207b267137d240adc5b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:08.915964 6779 consensus_queue.cc:1035] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.916816 5839 tablet_bootstrap.cc:654] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:08.923278 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a59c7257e74b4abaa4ea7a1ee9136d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:08.923537 5440 raft_consensus.cc:3037] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:08.927039 5440 raft_consensus.cc:2454] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:08.926997 6913 consensus_queue.cc:1035] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:08.927759 6547 log.cc:1196] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/107d9b3f95e14044854ad0bf91a268ee
I20240507 08:12:08.928155 6547 ts_tablet_manager.cc:1909] T 107d9b3f95e14044854ad0bf91a268ee P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:08.929016 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 796) took 2461 ms. Trace:
I20240507 08:12:08.929956 6547 rpcz_store.cc:276] 0507 08:12:06.467317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.467360 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:08.859366 (+2392006us) ts_tablet_manager.cc:1120] Deleting tablet 107d9b3f95e14044854ad0bf91a268ee
0507 08:12:08.859375 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.889483 (+ 30108us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.893128 (+ 3645us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.928995 (+ 35867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2391872}
I20240507 08:12:08.930552 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 797) took 2357 ms. Trace:
I20240507 08:12:08.930720 5684 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:55948 (request call id 864) took 1317 ms. Trace:
I20240507 08:12:08.930812 5684 rpcz_store.cc:276] 0507 08:12:07.613432 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.725670 (+112238us) service_pool.cc:225] Handling call
0507 08:12:07.726263 (+ 593us) ts_tablet_manager.cc:642] Acquired tablet manager lock
0507 08:12:07.726311 (+ 48us) ts_tablet_manager.cc:655] Creating new metadata...
0507 08:12:08.785696 (+1059385us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.930570 (+144874us) tablet_replica.cc:205] Creating consensus instance
0507 08:12:08.930704 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20240507 08:12:08.930893 5562 tablet_service.cc:1467] Processing CreateTablet for tablet d026ebe0217b4e4faee751e50461d0e4 (DEFAULT_TABLE table=default.Table_af01a22cd534413c8b74f2bdda0c70d0 [id=5da7241b01394a08b844b74cd2fb05a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.931011 6547 rpcz_store.cc:276] 0507 08:12:06.572978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.573048 (+ 70us) service_pool.cc:225] Handling call
0507 08:12:08.930445 (+2357397us) ts_tablet_manager.cc:1120] Deleting tablet f294baf04bf74dfe8500cb8c1452d8b3
0507 08:12:08.930454 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.930540 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2357154}
I20240507 08:12:08.931267 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d026ebe0217b4e4faee751e50461d0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.931335 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 5370d022aa904900a28842b887b13187 (DEFAULT_TABLE table=default.Table_4ae0974a73994e749cf5734401312e03 [id=45ee1173fc914a238a573bedd7ca11ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.931707 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5370d022aa904900a28842b887b13187. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.938230 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 798) took 2232 ms. Trace:
I20240507 08:12:08.938727 6547 rpcz_store.cc:276] 0507 08:12:06.706093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.706165 (+ 72us) service_pool.cc:225] Handling call
0507 08:12:08.938112 (+2231947us) ts_tablet_manager.cc:1120] Deleting tablet 150efae7834740b798af86ae1dcfe445
0507 08:12:08.938120 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.938213 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231685}
I20240507 08:12:08.939458 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 799) took 2223 ms. Trace:
I20240507 08:12:08.939540 5700 tablet_service.cc:1467] Processing CreateTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.940373 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.941202 6547 rpcz_store.cc:276] 0507 08:12:06.716408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.716467 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:08.939361 (+2222894us) ts_tablet_manager.cc:1120] Deleting tablet 242ea35c61404a91b1211c5ad9207728
0507 08:12:08.939371 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.939444 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2222688}
I20240507 08:12:08.941722 6547 tablet_replica.cc:330] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:08.941942 6547 raft_consensus.cc:2229] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:08.942131 6547 raft_consensus.cc:2258] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:08.943302 5840 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:08.943428 5840 ts_tablet_manager.cc:1367] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.114s user 0.002s sys 0.000s
I20240507 08:12:08.944123 5840 raft_consensus.cc:348] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.944327 5840 raft_consensus.cc:374] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.944391 5840 raft_consensus.cc:729] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.944633 5693 tablet_service.cc:1467] Processing CreateTablet for tablet d07221497ce7468fb86c257492cef397 (DEFAULT_TABLE table=default.Table_5bbaa170e90d41659516afd31247ef2d [id=a16594053e1e457094c727fef006564b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.945012 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07221497ce7468fb86c257492cef397. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.946468 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 (DEFAULT_TABLE table=default.Table_760b66fcd51f489d8279e16c09e9955e [id=a3fd2f8ee2c4431b8e6d08b612bfb2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.946496 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 107d9b3f95e14044854ad0bf91a268ee (table default.Table_351291351f3c4972ac5f9cd896d68312 [id=ccea3f017cc84cce95f3010023a76bb4]) successfully deleted
I20240507 08:12:08.946827 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e5e67df91f4029bf55e5a82dc31f82. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:08.947170 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f294baf04bf74dfe8500cb8c1452d8b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f294baf04bf74dfe8500cb8c1452d8b3 already in progress: creating tablet
W20240507 08:12:08.948668 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a25b41cf0d45f9bc3b2d9731a10da3 consensus state has no leader
I20240507 08:12:08.948984 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cf86bb5af528497597aa0f0affb3524e" } modify_external_catalogs: true
I20240507 08:12:08.949195 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cf86bb5af528497597aa0f0affb3524e" } modify_external_catalogs: true
W20240507 08:12:08.949672 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be1b36ca5b6041d780e851f137d3a4c9 consensus state has no leader
I20240507 08:12:08.949733 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:08.949998 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_af01a22cd534413c8b74f2bdda0c70d0" } new_table_name: "default.Table_05b8d7948bd44fa68a4b79264fecdda6" modify_external_catalogs: true
I20240507 08:12:08.950253 5681 tablet_service.cc:1467] Processing CreateTablet for tablet a905e345412541558b30c629a2eb86c5 (DEFAULT_TABLE table=default.Table_1d55b33508cc4645b468556b1384ffda [id=469a5ccf197d4ac4a03e1ca53d61d528]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.950670 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8e9ea4672e9e4edcab31a6d54cd933e3" } new_table_name: "default.Table_92a943da9da941f9aa6f4dd810a2ec6d" modify_external_catalogs: true
W20240507 08:12:08.951277 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe721ba2f0ed4a3a887e7496e85d163c consensus state has no leader
I20240507 08:12:08.951567 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5340a70e2ea3467e844c940b4b41d9ee" } modify_external_catalogs: true
I20240507 08:12:08.951701 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5340a70e2ea3467e844c940b4b41d9ee" } modify_external_catalogs: true
I20240507 08:12:08.955072 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 182124e13ca749c8a2a96d38514158b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955220 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955325 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 4716c1285e91432a92b25ab13ada2779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955439 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955540 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955636 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955729 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955821 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.955919 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet d2240fd14155495a8c97a0af96090faf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.956030 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fc65070bbae4db19c5d14b6670d599c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.956127 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet ec3e1823f9ff4f19be2b13cdcd706f59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.956245 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b0c793dbc884efd92b49ae7798206a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:08.956226 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e7eef600626a4961ab1fae27162537bb
I20240507 08:12:08.956553 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 76c06edeabdd462292ca8c23d9cba62f (DEFAULT_TABLE table=default.Table_9b7222e8f81c442098889182635a4503 [id=8c8e7c490a284e8cb2ea7f301b1443fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.956869 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76c06edeabdd462292ca8c23d9cba62f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.956979 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c05cf821365c4b04a6992a5daf8cc22f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.957494 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c05cf821365c4b04a6992a5daf8cc22f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.959388 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bfa2139385d34c37a7ac14d78480a779
I20240507 08:12:08.960148 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 620214b5c5bc43c98df65932e1eed2db (DEFAULT_TABLE table=default.Table_b9b6ac9544f74a3f8ecc609196a1b255 [id=d529797cb7bc4d5fb0c2598e0b8b8e3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.960078 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a79ee8acf6384968a6fffcbb9284c2eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.960520 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a79ee8acf6384968a6fffcbb9284c2eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.962241 5843 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:08.962380 5843 ts_tablet_manager.cc:1367] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.219s user 0.005s sys 0.008s
I20240507 08:12:08.962604 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2139500f65f9422c8a1662c61b6bb1c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.963027 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2139500f65f9422c8a1662c61b6bb1c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.963043 5843 raft_consensus.cc:348] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.963225 5843 raft_consensus.cc:374] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:08.963276 5843 raft_consensus.cc:729] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:08.963717 5696 tablet_service.cc:1467] Processing CreateTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 (DEFAULT_TABLE table=default.Table_855c7a629b1c4b48a6859a9d4d812be6 [id=8d1b3ce281004cebaa989dadd5dee2fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.966279 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f2de26190bc44916809856d84ad68345
I20240507 08:12:08.966928 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f1ba0da75f11411ab9babc78e508f6e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.967439 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1ba0da75f11411ab9babc78e508f6e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.970110 6547 ts_tablet_manager.cc:1875] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:08.970984 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:08.972702 6936 log.cc:924] Time spent T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Append to log took a long time: real 0.283s user 0.000s sys 0.000s
I20240507 08:12:08.976163 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240507 08:12:08.976406 5687 tablet_service.cc:1467] Processing CreateTablet for tablet d026ebe0217b4e4faee751e50461d0e4 (DEFAULT_TABLE table=default.Table_af01a22cd534413c8b74f2bdda0c70d0 [id=5da7241b01394a08b844b74cd2fb05a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.977238 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2365998a0bb04c22bea71d6a205a57cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:08.977319 6959 tablet.cc:1722] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240507 08:12:08.977841 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2365998a0bb04c22bea71d6a205a57cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:08.978726 5378 consensus_peers.cc:485] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 077ccc3946b34008bea7b80d8aee2662. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:08.983110 5841 ts_tablet_manager.cc:1888] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:08.983361 5841 log.cc:1196] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4716c1285e91432a92b25ab13ada2779
W20240507 08:12:08.983956 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 150efae7834740b798af86ae1dcfe445 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 150efae7834740b798af86ae1dcfe445 already in progress: creating tablet
W20240507 08:12:08.984308 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 242ea35c61404a91b1211c5ad9207728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242ea35c61404a91b1211c5ad9207728 already in progress: creating tablet
I20240507 08:12:08.984867 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cd9885f1f58049c599b7710b4e1f8977"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:08.985395 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd9885f1f58049c599b7710b4e1f8977 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:08.987188 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 03c2be5fbb2743daad3e6db48bd3996d (DEFAULT_TABLE table=default.Table_13bcd30a99834ef49f750a9bb990f583 [id=c346662349e84140b4b7d216a7496e58]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:08.988142 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a905e345412541558b30c629a2eb86c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:08.944557 5840 consensus_queue.cc:260] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:08.998129 5840 ts_tablet_manager.cc:1398] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.055s user 0.001s sys 0.001s
I20240507 08:12:08.998500 5840 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.000095 5840 tablet_bootstrap.cc:654] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.002279 6604 fs_manager.cc:725] Time spent opening block manager: real 0.208s user 0.006s sys 0.001s
I20240507 08:12:09.002542 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data,/tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
uuid: "a39de2a69ed4492aab4d7a232b09c1ae"
format_stamp: "Formatted at 2024-05-07 08:11:46 on dist-test-slave-d8w8"
I20240507 08:12:09.003854 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20240507 08:12:09.012118 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b95711fb20684223a02b09114419ce9a consensus state has no leader
I20240507 08:12:09.012526 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 35e34b60b78b402faf51e003c97db787 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:09.012701 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d94eb57a25bc440196abe6861ca19b95
I20240507 08:12:09.013077 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet fccc127d380d4035905c1b152ab1017b from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:09.018260 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d026ebe0217b4e4faee751e50461d0e4 consensus state has no leader
I20240507 08:12:09.035689 5841 ts_tablet_manager.cc:1909] T 4716c1285e91432a92b25ab13ada2779 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:09.023631 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.039623 5439 raft_consensus.cc:2454] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:09.024382 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b4b84ab3f274185a2b838ebb969ab52" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.039991 5438 raft_consensus.cc:3037] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.025094 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ba2e8d1e01047fd9cc1c192e9628750" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.040511 5445 raft_consensus.cc:3037] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.042739 5438 raft_consensus.cc:2454] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.042855 5445 raft_consensus.cc:2454] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.025444 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a3e58ba8c26d4f17af24eb379aca1f60" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.043043 5443 raft_consensus.cc:3037] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.043774 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 750) took 4321 ms. Trace:
I20240507 08:12:09.043844 5841 rpcz_store.cc:276] 0507 08:12:04.722315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.722789 (+ 474us) service_pool.cc:225] Handling call
0507 08:12:08.836646 (+4113857us) ts_tablet_manager.cc:1120] Deleting tablet 4716c1285e91432a92b25ab13ada2779
0507 08:12:08.836653 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.911557 (+ 74904us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.982996 (+ 71439us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.043749 (+ 60753us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4113578}
I20240507 08:12:09.044137 5841 tablet_replica.cc:330] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:09.045837 5443 raft_consensus.cc:2454] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.028826 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b5e3d23bbb6439a87b303dff1b44a1d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.046357 5440 raft_consensus.cc:3037] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.050875 5440 raft_consensus.cc:2454] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.051883 5840 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.052044 5840 ts_tablet_manager.cc:1367] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.002s
I20240507 08:12:09.012212 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 (DEFAULT_TABLE table=default.Table_760b66fcd51f489d8279e16c09e9955e [id=a3fd2f8ee2c4431b8e6d08b612bfb2ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.016314 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 620214b5c5bc43c98df65932e1eed2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.055712 5840 raft_consensus.cc:348] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.055899 5840 raft_consensus.cc:374] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.055961 5840 raft_consensus.cc:729] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.056159 5840 consensus_queue.cc:260] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.056682 5840 ts_tablet_manager.cc:1398] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20240507 08:12:09.056943 5840 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.058364 5840 tablet_bootstrap.cc:654] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.058560 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d026ebe0217b4e4faee751e50461d0e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.061666 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03c2be5fbb2743daad3e6db48bd3996d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.064369 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ad4ed7878454fea930b3982bcb6eed4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.064901 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c1932ffe7d37455c8aa6bb83ffd11498. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.068851 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet e7eef600626a4961ab1fae27162537bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:09.039208 6955 tablet.cc:1722] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:09.053148 5841 raft_consensus.cc:2229] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.071492 5841 raft_consensus.cc:2258] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.073679 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de26190bc44916809856d84ad68345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:09.074021 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet d94eb57a25bc440196abe6861ca19b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:09.074471 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet e7eef600626a4961ab1fae27162537bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:09.074700 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa2139385d34c37a7ac14d78480a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:09.076647 5262 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 169626f9a0bc4c6f9d8be11c0a802816 (table default.Table_4dbb2a240d8349e8b24875b8f9e99eab [id=ff9fb60fc9774237a4515c33f857fbf1]). Expected version 4 got 0
I20240507 08:12:09.089816 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet bd6ca1a20df748e097de097d1c2d7be3 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:09.093417 5841 ts_tablet_manager.cc:1875] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.093395 6547 ts_tablet_manager.cc:1888] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:09.093534 6547 log.cc:1196] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4fcc1d7e309946d783a524b61d5f1269
I20240507 08:12:09.093989 6547 ts_tablet_manager.cc:1909] T 4fcc1d7e309946d783a524b61d5f1269 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.096629 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 800) took 2359 ms. Trace:
I20240507 08:12:09.096719 6547 rpcz_store.cc:276] 0507 08:12:06.737216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.737892 (+ 676us) service_pool.cc:225] Handling call
0507 08:12:08.941682 (+2203790us) ts_tablet_manager.cc:1120] Deleting tablet 4fcc1d7e309946d783a524b61d5f1269
0507 08:12:08.941691 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.073688 (+131997us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.093278 (+ 19590us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.096611 (+ 3333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203543}
I20240507 08:12:09.101083 6547 tablet_replica.cc:330] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.101370 6547 raft_consensus.cc:2229] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.101567 6547 raft_consensus.cc:2258] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.106020 6547 ts_tablet_manager.cc:1875] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.118503 5839 tablet_bootstrap.cc:492] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.118636 5839 ts_tablet_manager.cc:1367] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.209s user 0.020s sys 0.011s
I20240507 08:12:09.119318 5839 raft_consensus.cc:348] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.119513 6547 ts_tablet_manager.cc:1888] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.119553 5839 raft_consensus.cc:374] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.119593 6547 log.cc:1196] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5089db17ca904a04878d83209001c54e
I20240507 08:12:09.119612 5839 raft_consensus.cc:729] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.119882 6547 ts_tablet_manager.cc:1909] T 5089db17ca904a04878d83209001c54e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.119828 5839 consensus_queue.cc:260] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.120157 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 801) took 2328 ms. Trace:
I20240507 08:12:09.120237 6547 rpcz_store.cc:276] 0507 08:12:06.791808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.791872 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:09.096980 (+2305108us) ts_tablet_manager.cc:1120] Deleting tablet 5089db17ca904a04878d83209001c54e
0507 08:12:09.096989 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.115315 (+ 18326us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.119432 (+ 4117us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.120136 (+ 704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2304853}
I20240507 08:12:09.120379 5839 ts_tablet_manager.cc:1398] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:09.120548 6547 tablet_replica.cc:330] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.120630 5839 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.120708 6547 raft_consensus.cc:2229] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.120826 6547 raft_consensus.cc:2258] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.121256 6547 ts_tablet_manager.cc:1875] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.127925 5839 tablet_bootstrap.cc:654] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.130048 6547 ts_tablet_manager.cc:1888] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.130146 6547 log.cc:1196] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1cd02a49359743c4b4e540ab0338d8ed
I20240507 08:12:09.130426 6547 ts_tablet_manager.cc:1909] T 1cd02a49359743c4b4e540ab0338d8ed P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.130431 5839 tablet_bootstrap.cc:492] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.130579 5839 ts_tablet_manager.cc:1367] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240507 08:12:09.130745 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 802) took 2338 ms. Trace:
I20240507 08:12:09.130832 6547 rpcz_store.cc:276] 0507 08:12:06.792408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.792460 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:09.120507 (+2328047us) ts_tablet_manager.cc:1120] Deleting tablet 1cd02a49359743c4b4e540ab0338d8ed
0507 08:12:09.120515 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.126887 (+ 6372us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.129979 (+ 3092us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.130727 (+ 748us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2327829}
I20240507 08:12:09.131316 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 804) took 2258 ms. Trace:
I20240507 08:12:09.131393 6547 rpcz_store.cc:276] 0507 08:12:06.872423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.872494 (+ 71us) service_pool.cc:225] Handling call
0507 08:12:09.131202 (+2258708us) ts_tablet_manager.cc:1120] Deleting tablet 96d7d8bcb635416693395b1f9ad27e2f
0507 08:12:09.131212 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.131301 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258479}
I20240507 08:12:09.131359 5839 raft_consensus.cc:348] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.131556 5839 raft_consensus.cc:374] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.131618 5839 raft_consensus.cc:729] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.131923 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 806) took 2143 ms. Trace:
I20240507 08:12:09.132010 6547 rpcz_store.cc:276] 0507 08:12:06.988837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.988888 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:09.131831 (+2142943us) ts_tablet_manager.cc:1120] Deleting tablet abd47fb41bd14411bb44cc0f25e8cac4
0507 08:12:09.131842 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.131910 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2142705}
I20240507 08:12:09.131846 5839 consensus_queue.cc:260] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.132665 5839 ts_tablet_manager.cc:1398] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.133742 5839 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.135012 5839 tablet_bootstrap.cc:654] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.138203 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 813) took 2083 ms. Trace:
I20240507 08:12:09.138295 6547 rpcz_store.cc:276] 0507 08:12:07.054369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.056560 (+ 2191us) service_pool.cc:225] Handling call
0507 08:12:09.132834 (+2076274us) ts_tablet_manager.cc:1120] Deleting tablet aa485207d0f140c99a9386cb4002c480
0507 08:12:09.138108 (+ 5274us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.138187 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2076106}
I20240507 08:12:09.138923 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 825) took 1946 ms. Trace:
I20240507 08:12:09.138937 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet d94eb57a25bc440196abe6861ca19b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:09.139006 6547 rpcz_store.cc:276] 0507 08:12:07.192804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.192864 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:09.138802 (+1945938us) ts_tablet_manager.cc:1120] Deleting tablet b60ec15a83934999871425ef9a64ec72
0507 08:12:09.138820 (+ 18us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.138909 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1945714}
I20240507 08:12:09.139312 6547 tablet_replica.cc:330] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.139470 6547 raft_consensus.cc:2229] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.139583 6547 raft_consensus.cc:2258] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.070362 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa2139385d34c37a7ac14d78480a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:09.070637 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de26190bc44916809856d84ad68345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:09.098495 5840 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.140092 5840 ts_tablet_manager.cc:1367] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.001s
I20240507 08:12:09.140736 5840 raft_consensus.cc:348] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.140913 5840 raft_consensus.cc:374] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.140957 5840 raft_consensus.cc:729] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.141120 5840 consensus_queue.cc:260] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.141527 5840 ts_tablet_manager.cc:1398] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240507 08:12:09.141834 5840 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.142704 5656 leader_election.cc:302] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.143113 6780 raft_consensus.cc:2781] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.143246 5840 tablet_bootstrap.cc:654] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.143249 6780 raft_consensus.cc:686] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.143666 6780 consensus_queue.cc:237] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.152436 6547 ts_tablet_manager.cc:1875] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.156594 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d553689c30b44cef8f22ed8aa85e9873"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:09.157147 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d553689c30b44cef8f22ed8aa85e9873 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:09.158354 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4a583347a22943338c2cad0211f8f17e" } new_table_name: "default.Table_709cc5845fc74e038578ef026cbb7567" modify_external_catalogs: true
I20240507 08:12:09.161743 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_afe14a8a082943dd9b05964cec987540" } new_table_name: "default.Table_68ee453328094fdcbff35893004c55b1" modify_external_catalogs: true
I20240507 08:12:09.164059 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a48b5e1d22504c999c0df606f5092147" } new_table_name: "default.Table_727fd7e435fd40368f63b993db76af91" modify_external_catalogs: true
I20240507 08:12:09.165740 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ca80faa43124b57991879a94bd046cd" } modify_external_catalogs: true
I20240507 08:12:09.165942 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3ca80faa43124b57991879a94bd046cd" } modify_external_catalogs: true
I20240507 08:12:09.165977 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e07a25a325a64352a0433eee53f843ab" } modify_external_catalogs: true
I20240507 08:12:09.166188 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e07a25a325a64352a0433eee53f843ab" } modify_external_catalogs: true
I20240507 08:12:09.172019 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5ff9542c98ae41fe80e979ae729f94cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:09.172756 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ff9542c98ae41fe80e979ae729f94cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:09.172744 6946 raft_consensus.cc:482] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.173003 6946 raft_consensus.cc:504] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.173681 6946 leader_election.cc:288] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:09.000324 5843 consensus_queue.cc:260] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.174610 6880 raft_consensus.cc:482] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.174587 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:09.174724 6880 raft_consensus.cc:504] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.175128 5378 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.175545 6880 leader_election.cc:288] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:09.176106 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.176208 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.176648 5377 leader_election.cc:341] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:09.176838 5378 leader_election.cc:341] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.176921 5378 leader_election.cc:302] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.177208 5840 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.177237 6880 raft_consensus.cc:2726] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.177333 5840 ts_tablet_manager.cc:1367] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.000s
I20240507 08:12:09.177989 5840 raft_consensus.cc:348] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.178098 5843 ts_tablet_manager.cc:1398] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.216s user 0.001s sys 0.001s
I20240507 08:12:09.178179 5840 raft_consensus.cc:374] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.178231 5840 raft_consensus.cc:729] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.178416 5840 consensus_queue.cc:260] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.178961 5840 ts_tablet_manager.cc:1398] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:09.179348 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.179875 5840 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.181450 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4716c1285e91432a92b25ab13ada2779 (table default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb [id=bd83f9b19311401a8ff3fd39e8c75e8f]) successfully deleted
I20240507 08:12:09.182422 5840 tablet_bootstrap.cc:654] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:09.180492 5377 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.197929 5377 leader_election.cc:302] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.182699 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet e7eef600626a4961ab1fae27162537bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:09.182929 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de26190bc44916809856d84ad68345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:09.182987 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet d94eb57a25bc440196abe6861ca19b95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:09.183157 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet bfa2139385d34c37a7ac14d78480a779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:09.198675 6880 raft_consensus.cc:2726] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.199564 5843 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.199594 6979 tablet.cc:1722] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:09.203437 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e5e67df91f4029bf55e5a82dc31f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.207697 5840 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.207854 5840 ts_tablet_manager.cc:1367] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240507 08:12:09.208635 5840 raft_consensus.cc:348] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.208887 5840 raft_consensus.cc:374] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.208940 5840 raft_consensus.cc:729] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.212921 5840 consensus_queue.cc:260] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.213547 5840 ts_tablet_manager.cc:1398] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:09.213804 5840 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.215255 5840 tablet_bootstrap.cc:654] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.221895 6547 ts_tablet_manager.cc:1888] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.222003 6547 log.cc:1196] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/91ee3a6c4ac642a3921560ea13bc1341
I20240507 08:12:09.222292 6547 ts_tablet_manager.cc:1909] T 91ee3a6c4ac642a3921560ea13bc1341 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.222640 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 826) took 1855 ms. Trace:
I20240507 08:12:09.223469 6880 raft_consensus.cc:482] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.223604 6880 raft_consensus.cc:504] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.224162 6880 leader_election.cc:288] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:09.225224 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:09.225929 5377 leader_election.cc:341] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:09.228878 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 86211.5us
I20240507 08:12:09.229014 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.229503 5656 consensus_peers.cc:485] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b4504b1ed3d449619926fe80469d9790. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:09.230055 5378 leader_election.cc:341] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.230165 5378 leader_election.cc:302] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.230600 6880 raft_consensus.cc:2726] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.230603 5656 leader_election.cc:302] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.230815 6784 raft_consensus.cc:2781] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.230901 6784 raft_consensus.cc:482] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.230962 6784 raft_consensus.cc:3037] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.222718 6547 rpcz_store.cc:276] 0507 08:12:07.366956 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.367047 (+ 91us) service_pool.cc:225] Handling call
0507 08:12:09.139277 (+1772230us) ts_tablet_manager.cc:1120] Deleting tablet 91ee3a6c4ac642a3921560ea13bc1341
0507 08:12:09.139284 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.199375 (+ 60091us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.221795 (+ 22420us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.222622 (+ 827us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771963}
I20240507 08:12:09.231252 6547 tablet_replica.cc:330] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.231429 6547 raft_consensus.cc:2229] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.231560 6547 raft_consensus.cc:2258] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.231917 5656 leader_election.cc:302] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.232041 6547 ts_tablet_manager.cc:1875] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.232244 6675 raft_consensus.cc:2781] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.233028 5839 tablet_bootstrap.cc:492] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.233129 5839 ts_tablet_manager.cc:1367] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.100s user 0.005s sys 0.001s
I20240507 08:12:09.233783 5839 raft_consensus.cc:348] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.233949 5839 raft_consensus.cc:374] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.233990 5839 raft_consensus.cc:729] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.234159 5839 consensus_queue.cc:260] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.244479 5839 ts_tablet_manager.cc:1398] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:09.247689 5843 tablet_bootstrap.cc:654] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.248193 6880 raft_consensus.cc:482] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.248320 6880 raft_consensus.cc:504] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.248983 6880 leader_election.cc:288] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:09.249519 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:09.249943 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:09.250043 5378 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:09.250533 5377 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.250638 5377 leader_election.cc:302] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.251017 6880 raft_consensus.cc:2726] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.252694 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 744) took 4246 ms. Trace:
I20240507 08:12:09.256425 5839 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.258028 5839 tablet_bootstrap.cc:654] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.258924 6513 rpcz_store.cc:276] 0507 08:12:05.006038 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.006415 (+ 377us) service_pool.cc:225] Handling call
0507 08:12:08.850381 (+3843966us) ts_tablet_manager.cc:1120] Deleting tablet c48da19c9b9e4207b267137d240adc5b
0507 08:12:08.850390 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:08.889291 (+ 38901us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:08.912404 (+ 23113us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.252673 (+340269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3843856}
I20240507 08:12:09.259425 6513 tablet_replica.cc:330] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:09.259557 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c48da19c9b9e4207b267137d240adc5b (table default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]) successfully deleted
I20240507 08:12:09.259653 5840 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.259768 5840 ts_tablet_manager.cc:1367] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.002s
I20240507 08:12:09.259773 6513 raft_consensus.cc:2229] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:09.260529 5840 raft_consensus.cc:348] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.260728 5840 raft_consensus.cc:374] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.260784 5840 raft_consensus.cc:729] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.261021 6513 raft_consensus.cc:2258] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.260954 5840 consensus_queue.cc:260] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.263056 5840 ts_tablet_manager.cc:1398] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:12:09.263367 5840 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.263635 6912 raft_consensus.cc:482] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.263762 6912 raft_consensus.cc:504] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.264370 6912 leader_election.cc:288] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.264576 5840 tablet_bootstrap.cc:654] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.264914 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027b4cefa4d41659f819ee4326470c4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.265415 5654 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.266444 6886 raft_consensus.cc:482] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.266568 6886 raft_consensus.cc:504] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.267284 6886 leader_election.cc:288] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:09.268043 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.270184 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:09.272433 5840 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.272567 5840 ts_tablet_manager.cc:1367] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:09.273226 5840 raft_consensus.cc:348] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.273438 5840 raft_consensus.cc:374] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.273506 5840 raft_consensus.cc:729] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.273756 5840 consensus_queue.cc:260] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.274228 5840 ts_tablet_manager.cc:1398] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:09.274492 5840 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.275697 5840 tablet_bootstrap.cc:654] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.277005 5839 tablet_bootstrap.cc:492] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.277124 5839 ts_tablet_manager.cc:1367] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20240507 08:12:09.277771 6547 ts_tablet_manager.cc:1888] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.277873 5839 raft_consensus.cc:348] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.278050 5839 raft_consensus.cc:374] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.278118 5839 raft_consensus.cc:729] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.277882 6547 log.cc:1196] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1b19d8e28f164c588946ea5e81ec320e
I20240507 08:12:09.278575 6547 ts_tablet_manager.cc:1909] T 1b19d8e28f164c588946ea5e81ec320e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.278846 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 827) took 1911 ms. Trace:
I20240507 08:12:09.278923 6547 rpcz_store.cc:276] 0507 08:12:07.367243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.367296 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:09.231209 (+1863913us) ts_tablet_manager.cc:1120] Deleting tablet 1b19d8e28f164c588946ea5e81ec320e
0507 08:12:09.231217 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.247724 (+ 16507us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.277684 (+ 29960us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.278831 (+ 1147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1863767}
I20240507 08:12:09.280328 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 828) took 1869 ms. Trace:
I20240507 08:12:09.280408 6547 rpcz_store.cc:276] 0507 08:12:07.411095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.431057 (+ 19962us) service_pool.cc:225] Handling call
0507 08:12:09.280221 (+1849164us) ts_tablet_manager.cc:1120] Deleting tablet 5f0b399a2d074f4ca3626ff17bd5518e
0507 08:12:09.280230 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.280313 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1848956}
I20240507 08:12:09.280762 5839 consensus_queue.cc:260] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.281129 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 830) took 1835 ms. Trace:
I20240507 08:12:09.284010 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4fcc1d7e309946d783a524b61d5f1269 (table default.Table_bef65763cc5e492f993abf19cd31e433 [id=07e673b316df458a93a171ccc5f59a55]) successfully deleted
I20240507 08:12:09.284402 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5089db17ca904a04878d83209001c54e (table default.Table_522ea36e321d413c977697e907f9e118 [id=01fe96300c824082a8d67b5c95036ba2]) successfully deleted
I20240507 08:12:09.284706 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1cd02a49359743c4b4e540ab0338d8ed (table default.Table_75283770e2c44851bcfa380f6385359d [id=fab372c29d234af7b0bfdae99be63919]) successfully deleted
W20240507 08:12:09.285092 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 96d7d8bcb635416693395b1f9ad27e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96d7d8bcb635416693395b1f9ad27e2f already in progress: creating tablet
W20240507 08:12:09.285413 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet abd47fb41bd14411bb44cc0f25e8cac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd47fb41bd14411bb44cc0f25e8cac4 already in progress: creating tablet
I20240507 08:12:09.287822 6547 rpcz_store.cc:276] 0507 08:12:07.445484 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.445542 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:09.281034 (+1835492us) ts_tablet_manager.cc:1120] Deleting tablet dfd9624f4337437bb11b247048e28c08
0507 08:12:09.281042 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.281117 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1835285}
I20240507 08:12:09.289497 6513 ts_tablet_manager.cc:1875] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:09.297883 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 65975.8us
I20240507 08:12:09.298605 5656 leader_election.cc:302] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.298954 6912 raft_consensus.cc:2781] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.299078 6912 raft_consensus.cc:686] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.299372 6912 consensus_queue.cc:237] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.301297 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 829) took 1889 ms. Trace:
I20240507 08:12:09.301398 6547 rpcz_store.cc:276] 0507 08:12:07.411366 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.464619 (+ 53253us) service_pool.cc:225] Handling call
0507 08:12:09.301162 (+1836543us) ts_tablet_manager.cc:1120] Deleting tablet 574a47894d274360891253534bf215a5
0507 08:12:09.301174 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.301278 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1836319}
I20240507 08:12:09.301813 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 833) took 1778 ms. Trace:
I20240507 08:12:09.301894 6547 rpcz_store.cc:276] 0507 08:12:07.523656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.523719 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:09.301701 (+1777982us) ts_tablet_manager.cc:1120] Deleting tablet 5aa521dd21634cd0af25652931069c41
0507 08:12:09.301710 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.301798 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1777720}
I20240507 08:12:09.302224 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 843) took 1645 ms. Trace:
I20240507 08:12:09.302260 5840 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.302296 6547 rpcz_store.cc:276] 0507 08:12:07.657117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.658231 (+ 1114us) service_pool.cc:225] Handling call
0507 08:12:09.302120 (+1643889us) ts_tablet_manager.cc:1120] Deleting tablet fccc127d380d4035905c1b152ab1017b
0507 08:12:09.302128 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.302211 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1643636}
I20240507 08:12:09.302408 5840 ts_tablet_manager.cc:1367] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.006s
I20240507 08:12:09.302628 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 844) took 1611 ms. Trace:
I20240507 08:12:09.302699 6547 rpcz_store.cc:276] 0507 08:12:07.691176 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.691261 (+ 85us) service_pool.cc:225] Handling call
0507 08:12:09.302524 (+1611263us) ts_tablet_manager.cc:1120] Deleting tablet 32ed504c4cf443e48ab285466b9e9b5d
0507 08:12:09.302532 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.302613 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1605945}
I20240507 08:12:09.302994 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 845) took 1611 ms. Trace:
I20240507 08:12:09.303063 6547 rpcz_store.cc:276] 0507 08:12:07.691286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.720813 (+ 29527us) service_pool.cc:225] Handling call
0507 08:12:09.302906 (+1582093us) ts_tablet_manager.cc:1120] Deleting tablet bc029bc110b847f79d876ec3ab568832
0507 08:12:09.302915 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.302979 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1581819}
I20240507 08:12:09.303347 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 864) took 1047 ms. Trace:
I20240507 08:12:09.303427 6547 rpcz_store.cc:276] 0507 08:12:08.255538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.255618 (+ 80us) service_pool.cc:225] Handling call
0507 08:12:09.303263 (+1047645us) ts_tablet_manager.cc:1120] Deleting tablet 8005100f64fd49a3ac29d34aa728fdfb
0507 08:12:09.303271 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.303334 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1047388}
I20240507 08:12:09.303738 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 865) took 1035 ms. Trace:
I20240507 08:12:09.303809 6547 rpcz_store.cc:276] 0507 08:12:08.268089 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.268146 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:09.303631 (+1035485us) ts_tablet_manager.cc:1120] Deleting tablet a570e39a52d84d3dbcaeb9ea005d9f41
0507 08:12:09.303638 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.303724 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1035297}
I20240507 08:12:09.303818 5840 raft_consensus.cc:348] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.304025 5840 raft_consensus.cc:374] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.304082 5840 raft_consensus.cc:729] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.304121 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 866) took 1025 ms. Trace:
I20240507 08:12:09.304193 6547 rpcz_store.cc:276] 0507 08:12:08.278631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.278683 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:09.304013 (+1025330us) ts_tablet_manager.cc:1120] Deleting tablet c40ed1afee294143ab85a8c03d080b74
0507 08:12:09.304020 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.304106 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025143}
I20240507 08:12:09.304332 5840 consensus_queue.cc:260] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.304543 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 867) took 1005 ms. Trace:
I20240507 08:12:09.304615 6547 rpcz_store.cc:276] 0507 08:12:08.298962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.299026 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:09.304443 (+1005417us) ts_tablet_manager.cc:1120] Deleting tablet c7bb96bb9d8f4930b531a3dd30d733c8
0507 08:12:09.304451 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.304529 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005193}
I20240507 08:12:09.304873 6547 tablet_replica.cc:330] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.304896 5840 ts_tablet_manager.cc:1398] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:09.305071 6547 raft_consensus.cc:2229] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.305181 5840 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.305248 6547 raft_consensus.cc:2258] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.287748 5839 ts_tablet_manager.cc:1398] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:09.305961 5839 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.306565 5840 tablet_bootstrap.cc:654] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.307543 5839 tablet_bootstrap.cc:654] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.308564 6547 ts_tablet_manager.cc:1875] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.316753 5843 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:09.322676 6547 ts_tablet_manager.cc:1888] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.322806 6547 log.cc:1196] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d537bed29a2f47058255d2530b5c0d23
I20240507 08:12:09.323125 6547 ts_tablet_manager.cc:1909] T d537bed29a2f47058255d2530b5c0d23 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.323586 6547 tablet_replica.cc:330] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.323762 6547 raft_consensus.cc:2229] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.323889 6547 raft_consensus.cc:2258] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.324398 6547 ts_tablet_manager.cc:1875] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.325528 6784 raft_consensus.cc:504] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.326727 6784 leader_election.cc:288] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.327374 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:09.328001 5654 leader_election.cc:341] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.328488 6784 raft_consensus.cc:482] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.328624 6784 raft_consensus.cc:504] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.329705 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.330224 5654 leader_election.cc:341] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.330566 6784 leader_election.cc:288] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.331677 5839 tablet_bootstrap.cc:492] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.331892 5839 ts_tablet_manager.cc:1367] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.000s
I20240507 08:12:09.332540 5839 raft_consensus.cc:348] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.332720 5839 raft_consensus.cc:374] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.332779 5839 raft_consensus.cc:729] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.332985 5839 consensus_queue.cc:260] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.336917 5839 ts_tablet_manager.cc:1398] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:09.338286 6547 ts_tablet_manager.cc:1888] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.338402 6547 log.cc:1196] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/2612c023740644cb9564c44f7bfe8b61
I20240507 08:12:09.338701 6547 ts_tablet_manager.cc:1909] T 2612c023740644cb9564c44f7bfe8b61 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:09.339007 6547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 870) took 1004 ms. Trace:
I20240507 08:12:09.339090 6547 rpcz_store.cc:276] 0507 08:12:08.334219 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.334266 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:09.323544 (+989278us) ts_tablet_manager.cc:1120] Deleting tablet 2612c023740644cb9564c44f7bfe8b61
0507 08:12:09.323552 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.335323 (+ 11771us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.338201 (+ 2878us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.338991 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":989128}
I20240507 08:12:09.340243 5839 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.344347 6547 tablet_replica.cc:330] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.344517 6547 raft_consensus.cc:2229] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.344645 6547 raft_consensus.cc:2258] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:09.344852 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 46270.6us
I20240507 08:12:09.345101 6547 ts_tablet_manager.cc:1875] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.347239 5656 leader_election.cc:302] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.358774 6547 ts_tablet_manager.cc:1888] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.358954 6547 log.cc:1196] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e1f9a56a95074e67bb33ceee1af48a18
I20240507 08:12:09.359265 6547 ts_tablet_manager.cc:1909] T e1f9a56a95074e67bb33ceee1af48a18 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:09.366700 5249 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 81260.5us
I20240507 08:12:09.367309 5843 ts_tablet_manager.cc:1367] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.002s
I20240507 08:12:09.369027 5839 tablet_bootstrap.cc:654] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.371963 6784 raft_consensus.cc:2781] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.372112 6784 raft_consensus.cc:686] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.372406 6784 consensus_queue.cc:237] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.373044 5839 tablet_bootstrap.cc:492] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.373162 5839 ts_tablet_manager.cc:1367] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.033s user 0.017s sys 0.001s
I20240507 08:12:09.373777 5839 raft_consensus.cc:348] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.373972 5839 raft_consensus.cc:374] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.374028 5839 raft_consensus.cc:729] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.374248 5839 consensus_queue.cc:260] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.378074 5839 ts_tablet_manager.cc:1398] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:09.380549 5656 leader_election.cc:302] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.381088 5839 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.381155 6913 raft_consensus.cc:2781] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.381280 6913 raft_consensus.cc:686] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.381702 6913 consensus_queue.cc:237] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.382428 5839 tablet_bootstrap.cc:654] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.389942 5839 tablet_bootstrap.cc:492] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.390071 5839 ts_tablet_manager.cc:1367] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20240507 08:12:09.390658 5839 raft_consensus.cc:348] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.390831 5839 raft_consensus.cc:374] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.390887 5839 raft_consensus.cc:729] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.391112 5839 consensus_queue.cc:260] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.393335 6547 tablet_replica.cc:330] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:09.393530 6547 raft_consensus.cc:2229] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.394440 6547 raft_consensus.cc:2258] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.394663 5839 ts_tablet_manager.cc:1398] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:09.395133 6547 ts_tablet_manager.cc:1875] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.395615 6959 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:55948 (request call id 893) took 1033 ms. Trace:
I20240507 08:12:09.397891 5839 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:12:09.398689 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aa485207d0f140c99a9386cb4002c480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa485207d0f140c99a9386cb4002c480 already in progress: creating tablet
I20240507 08:12:09.399134 5839 tablet_bootstrap.cc:654] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:09.401981 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b60ec15a83934999871425ef9a64ec72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b60ec15a83934999871425ef9a64ec72 already in progress: creating tablet
I20240507 08:12:09.402529 5839 tablet_bootstrap.cc:492] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.402644 5839 ts_tablet_manager.cc:1367] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240507 08:12:09.403288 5839 raft_consensus.cc:348] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.403455 5839 raft_consensus.cc:374] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.403503 5839 raft_consensus.cc:729] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.403703 5839 consensus_queue.cc:260] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.405356 5656 consensus_peers.cc:485] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:09.407310 5839 ts_tablet_manager.cc:1398] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:09.409037 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 91ee3a6c4ac642a3921560ea13bc1341 (table default.Table_bbe018a0c26f42e2988b531cbf4df1d2 [id=2594204f90a24811b32f6e53b08306ee]) successfully deleted
I20240507 08:12:09.410327 5839 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.411782 5839 tablet_bootstrap.cc:654] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.413506 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1b19d8e28f164c588946ea5e81ec320e (table default.Table_b47f75826f1a465a9c38010bdf4d65ce [id=a53fe70a96fe4a809952dbc6cdc3a7fe]) successfully deleted
I20240507 08:12:09.396723 5843 raft_consensus.cc:348] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.402237 6959 rpcz_store.cc:276] 0507 08:12:08.362495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.482228 (+119733us) service_pool.cc:225] Handling call
0507 08:12:09.395597 (+913369us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:08.483257 (+ 0us) alter_schema_op.cc:96] PREPARE ALTER-SCHEMA: Starting
0507 08:12:08.483350 (+ 93us) alter_schema_op.cc:57] Acquiring schema lock in exclusive mode
0507 08:12:08.483362 (+ 12us) alter_schema_op.cc:59] Acquired schema lock
0507 08:12:08.489883 (+ 6521us) alter_schema_op.cc:111] PREPARE ALTER-SCHEMA: finished
0507 08:12:08.489938 (+ 55us) alter_schema_op.cc:119] START. Timestamp: 1715069528489828
0507 08:12:08.489943 (+ 5us) op_driver.cc:331] REPLICATION: starting
0507 08:12:08.490199 (+ 256us) log.cc:842] Serialized 173 byte log entry
0507 08:12:08.916362 (+426163us) op_driver.cc:444] REPLICATION: finished
0507 08:12:08.977298 (+ 60936us) alter_schema_op.cc:124] APPLY ALTER-SCHEMA: Starting
0507 08:12:08.977734 (+ 436us) log.cc:842] Serialized 5 byte log entry
0507 08:12:09.194310 (+216576us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.395528 (+201218us) log.cc:842] Serialized 14 byte log entry
0507 08:12:09.395573 (+ 45us) alter_schema_op.cc:65] Released schema lock
0507 08:12:09.395576 (+ 3us) alter_schema_op.cc:167] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":60871,"prepare.queue_time_us":679,"prepare.run_cpu_time_us":2081,"prepare.run_wall_time_us":18778,"raft.queue_time_us":443,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":281,"replication_time_us":426391,"thread_start_us":107102,"threads_started":2,"wal-append.queue_time_us":11432}]]}
I20240507 08:12:09.422883 5843 raft_consensus.cc:374] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.423061 5843 raft_consensus.cc:729] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.423465 5843 consensus_queue.cc:260] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.423986 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5f0b399a2d074f4ca3626ff17bd5518e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0b399a2d074f4ca3626ff17bd5518e already in progress: creating tablet
I20240507 08:12:09.421563 6547 ts_tablet_manager.cc:1888] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.429773 6547 log.cc:1196] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c48da19c9b9e4207b267137d240adc5b
I20240507 08:12:09.430135 6547 ts_tablet_manager.cc:1909] T c48da19c9b9e4207b267137d240adc5b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:09.433308 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet dfd9624f4337437bb11b247048e28c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd9624f4337437bb11b247048e28c08 already in progress: creating tablet
I20240507 08:12:09.445366 5843 ts_tablet_manager.cc:1398] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
W20240507 08:12:09.445755 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5aa521dd21634cd0af25652931069c41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5aa521dd21634cd0af25652931069c41 already in progress: creating tablet
I20240507 08:12:09.446122 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f8f4205f1f48abaf4fcd7113ad44a8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.451932 5442 raft_consensus.cc:3037] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.454782 5442 raft_consensus.cc:2454] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.446640 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.455194 5444 raft_consensus.cc:2454] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:09.456154 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet fccc127d380d4035905c1b152ab1017b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fccc127d380d4035905c1b152ab1017b already in progress: creating tablet
I20240507 08:12:09.457729 5841 ts_tablet_manager.cc:1888] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:09.469013 6604 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240507 08:12:09.470093 6604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240507 08:12:09.470367 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
W20240507 08:12:09.483922 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 32ed504c4cf443e48ab285466b9e9b5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32ed504c4cf443e48ab285466b9e9b5d already in progress: creating tablet
I20240507 08:12:09.485790 5841 log.cc:1196] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/9e447562e82347a584e39586500c27c2
I20240507 08:12:09.488590 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "169626f9a0bc4c6f9d8be11c0a802816" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:09.490124 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bc029bc110b847f79d876ec3ab568832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc029bc110b847f79d876ec3ab568832 already in progress: creating tablet
I20240507 08:12:09.490747 5841 ts_tablet_manager.cc:1909] T 9e447562e82347a584e39586500c27c2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:09.494324 5656 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7027b4cefa4d41659f819ee4326470c4
W20240507 08:12:09.495081 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a570e39a52d84d3dbcaeb9ea005d9f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a570e39a52d84d3dbcaeb9ea005d9f41 already in progress: creating tablet
I20240507 08:12:09.497817 5517 leader_election.cc:302] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.482781 5843 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.498354 6886 raft_consensus.cc:2726] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.499431 5839 tablet_bootstrap.cc:492] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.499564 5839 ts_tablet_manager.cc:1367] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.089s user 0.025s sys 0.055s
I20240507 08:12:09.499884 5843 tablet_bootstrap.cc:654] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.500335 5839 raft_consensus.cc:348] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.500577 5839 raft_consensus.cc:374] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.500638 5839 raft_consensus.cc:729] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.500872 5839 consensus_queue.cc:260] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.501331 5839 ts_tablet_manager.cc:1398] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:09.503019 5839 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.504515 5839 tablet_bootstrap.cc:654] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.494536 6784 raft_consensus.cc:482] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.505666 6784 raft_consensus.cc:504] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.506903 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.507588 5654 leader_election.cc:341] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.507984 6784 leader_election.cc:288] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.508953 5656 leader_election.cc:302] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.509725 6912 raft_consensus.cc:2726] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:09.510210 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c40ed1afee294143ab85a8c03d080b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40ed1afee294143ab85a8c03d080b74 already in progress: creating tablet
I20240507 08:12:09.510375 5839 tablet_bootstrap.cc:492] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.510509 5839 ts_tablet_manager.cc:1367] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.001s
W20240507 08:12:09.510689 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c7bb96bb9d8f4930b531a3dd30d733c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7bb96bb9d8f4930b531a3dd30d733c8 already in progress: creating tablet
I20240507 08:12:09.511021 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d537bed29a2f47058255d2530b5c0d23 (table default.Table_a0ac2efb18004de2977210793a4b8de7 [id=60c21801bcc74bc5b8b43856fd0b56c1]) successfully deleted
I20240507 08:12:09.511142 5839 raft_consensus.cc:348] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.511307 5839 raft_consensus.cc:374] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.511356 5839 raft_consensus.cc:729] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.511399 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 2612c023740644cb9564c44f7bfe8b61 (table default.Table_bfe2378493e64efb93444e140f410dfe [id=44cb0720d31d4cc09b0e65da042d169f]) successfully deleted
I20240507 08:12:09.511543 5839 consensus_queue.cc:260] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.512012 5839 ts_tablet_manager.cc:1398] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240507 08:12:09.512646 5839 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.511885 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e1f9a56a95074e67bb33ceee1af48a18 (table default.Table_b8268009436d4ce088ecbc9aae6f3fdc [id=7950c4b53ab94acda95b0d0b0e49302c]) successfully deleted
I20240507 08:12:09.513887 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 751) took 4785 ms. Trace:
I20240507 08:12:09.514045 5839 tablet_bootstrap.cc:654] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.514065 5841 rpcz_store.cc:276] 0507 08:12:04.728406 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.728715 (+ 309us) service_pool.cc:225] Handling call
0507 08:12:09.044098 (+4315383us) ts_tablet_manager.cc:1120] Deleting tablet 9e447562e82347a584e39586500c27c2
0507 08:12:09.044107 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.096410 (+ 52303us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.457616 (+361206us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.513858 (+ 56242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4315096}
I20240507 08:12:09.515187 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 753) took 4755 ms. Trace:
I20240507 08:12:09.515355 5841 rpcz_store.cc:276] 0507 08:12:04.759270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.763785 (+ 4515us) service_pool.cc:225] Handling call
0507 08:12:09.515024 (+4751239us) ts_tablet_manager.cc:1120] Deleting tablet 33875331045146a9a02559a0b37d5c6f
0507 08:12:09.515049 (+ 25us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.515168 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4751047}
I20240507 08:12:09.515973 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 754) took 4745 ms. Trace:
I20240507 08:12:09.516121 5841 rpcz_store.cc:276] 0507 08:12:04.771009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.771269 (+ 260us) service_pool.cc:225] Handling call
0507 08:12:09.515853 (+4744584us) ts_tablet_manager.cc:1120] Deleting tablet 63685e9ef255477db589fa572b087b57
0507 08:12:09.515865 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.515956 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4744346}
I20240507 08:12:09.516671 5841 tablet_replica.cc:330] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:09.516963 5841 raft_consensus.cc:2229] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.517201 5841 raft_consensus.cc:2258] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.517925 5841 ts_tablet_manager.cc:1875] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.523626 5656 leader_election.cc:302] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.524078 6912 raft_consensus.cc:2781] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.524224 6912 raft_consensus.cc:482] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.524415 6912 raft_consensus.cc:3037] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.534323 6912 raft_consensus.cc:504] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.535799 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:09.536489 5654 leader_election.cc:341] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.540197 5656 leader_election.cc:302] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.540563 6784 raft_consensus.cc:2781] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.540781 6784 raft_consensus.cc:686] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.541155 6784 consensus_queue.cc:237] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.543005 5839 tablet_bootstrap.cc:492] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.543128 5839 ts_tablet_manager.cc:1367] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.005s
I20240507 08:12:09.543756 5839 raft_consensus.cc:348] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.543936 5839 raft_consensus.cc:374] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.543987 5839 raft_consensus.cc:729] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.545617 5839 consensus_queue.cc:260] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.514822 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 9e447562e82347a584e39586500c27c2 (table default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]) successfully deleted
W20240507 08:12:09.547858 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 33875331045146a9a02559a0b37d5c6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33875331045146a9a02559a0b37d5c6f already in progress: creating tablet
W20240507 08:12:09.548162 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 63685e9ef255477db589fa572b087b57 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 63685e9ef255477db589fa572b087b57 already in progress: creating tablet
W20240507 08:12:09.548668 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 182124e13ca749c8a2a96d38514158b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182124e13ca749c8a2a96d38514158b1 already in progress: creating tablet
I20240507 08:12:09.549805 5839 ts_tablet_manager.cc:1398] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:09.497148 6513 ts_tablet_manager.cc:1888] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:09.550981 6513 log.cc:1196] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:12:09.503939 5840 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.551246 5840 ts_tablet_manager.cc:1367] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.246s user 0.031s sys 0.059s
I20240507 08:12:09.551568 6513 ts_tablet_manager.cc:1909] T 33a425a90f0449bcaaacb4c3f653d26c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:09.551898 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 746) took 4542 ms. Trace:
I20240507 08:12:09.552016 5840 raft_consensus.cc:348] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.552177 5840 raft_consensus.cc:374] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.552230 5840 raft_consensus.cc:729] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.552413 5840 consensus_queue.cc:260] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.552892 5840 ts_tablet_manager.cc:1398] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:09.553133 5840 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:09.548979 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d2240fd14155495a8c97a0af96090faf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2240fd14155495a8c97a0af96090faf already in progress: creating tablet
W20240507 08:12:09.553881 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8fc65070bbae4db19c5d14b6670d599c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc65070bbae4db19c5d14b6670d599c already in progress: creating tablet
W20240507 08:12:09.554174 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ec3e1823f9ff4f19be2b13cdcd706f59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec3e1823f9ff4f19be2b13cdcd706f59 already in progress: creating tablet
I20240507 08:12:09.554457 6508 raft_consensus.cc:482] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.554607 6508 raft_consensus.cc:504] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.554787 5840 tablet_bootstrap.cc:654] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.555581 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.556185 5654 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.556583 6508 leader_election.cc:288] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:09.559450 5656 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e555a89d340448bc852f2441ca850213
W20240507 08:12:09.560611 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5b0c793dbc884efd92b49ae7798206a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0c793dbc884efd92b49ae7798206a2 already in progress: creating tablet
W20240507 08:12:09.560986 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f2de26190bc44916809856d84ad68345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2de26190bc44916809856d84ad68345 already in progress: creating tablet
I20240507 08:12:09.561527 5656 leader_election.cc:302] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.561933 6508 raft_consensus.cc:2726] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.560154 6912 leader_election.cc:288] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.552029 6513 rpcz_store.cc:276] 0507 08:12:05.009856 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.009916 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:09.259341 (+4249425us) ts_tablet_manager.cc:1120] Deleting tablet 33a425a90f0449bcaaacb4c3f653d26c
0507 08:12:09.259350 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.293107 (+ 33757us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.497029 (+203922us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.551881 (+ 54852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4249255}
I20240507 08:12:09.562884 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 763) took 4240 ms. Trace:
I20240507 08:12:09.563030 6513 rpcz_store.cc:276] 0507 08:12:05.322228 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.322641 (+ 413us) service_pool.cc:225] Handling call
0507 08:12:09.562775 (+4240134us) ts_tablet_manager.cc:1120] Deleting tablet 3769eb670ff442e2ad2fa5b6a934512a
0507 08:12:09.562786 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.562869 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239946}
I20240507 08:12:09.564203 5839 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.562610 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 33a425a90f0449bcaaacb4c3f653d26c (table default.Table_81657f34e12b4ca0b2fd8ee26ab0e663 [id=b1f2776824d24099bbc0110c3da97c2e]) successfully deleted
W20240507 08:12:09.564903 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d94eb57a25bc440196abe6861ca19b95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d94eb57a25bc440196abe6861ca19b95 already in progress: creating tablet
I20240507 08:12:09.563342 6912 raft_consensus.cc:482] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.565748 5839 tablet_bootstrap.cc:654] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.565742 6912 raft_consensus.cc:504] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.566742 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:09.567132 5840 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.567258 5840 ts_tablet_manager.cc:1367] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.004s
W20240507 08:12:09.567281 5654 leader_election.cc:341] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.563684 6675 raft_consensus.cc:686] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.567677 6675 consensus_queue.cc:237] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.565474 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3769eb670ff442e2ad2fa5b6a934512a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3769eb670ff442e2ad2fa5b6a934512a already in progress: creating tablet
W20240507 08:12:09.569152 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e7eef600626a4961ab1fae27162537bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7eef600626a4961ab1fae27162537bb already in progress: creating tablet
I20240507 08:12:09.571206 5839 tablet_bootstrap.cc:492] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.571323 5839 ts_tablet_manager.cc:1367] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.001s
I20240507 08:12:09.572016 5839 raft_consensus.cc:348] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.572221 5839 raft_consensus.cc:374] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.572294 5839 raft_consensus.cc:729] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.567993 5840 raft_consensus.cc:348] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.572391 5840 raft_consensus.cc:374] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.572466 5840 raft_consensus.cc:729] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.572641 5840 consensus_queue.cc:260] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.573144 5840 ts_tablet_manager.cc:1398] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:09.573397 5840 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.568142 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 766) took 4233 ms. Trace:
I20240507 08:12:09.573920 6513 rpcz_store.cc:276] 0507 08:12:05.334871 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.335101 (+ 230us) service_pool.cc:225] Handling call
0507 08:12:09.568041 (+4232940us) ts_tablet_manager.cc:1120] Deleting tablet b6ba3397e33b4cc7805a25e7535a383a
0507 08:12:09.568049 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.568127 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4232751}
I20240507 08:12:09.574051 5839 consensus_queue.cc:260] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.574859 5839 ts_tablet_manager.cc:1398] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:12:09.574882 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 767) took 4239 ms. Trace:
I20240507 08:12:09.575035 5840 tablet_bootstrap.cc:654] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.575578 5839 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.575163 6513 rpcz_store.cc:276] 0507 08:12:05.335033 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.335333 (+ 300us) service_pool.cc:225] Handling call
0507 08:12:09.574781 (+4239448us) ts_tablet_manager.cc:1120] Deleting tablet 2bb648fe98e74248b5cd65603de58a02
0507 08:12:09.574789 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.574866 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4239314}
I20240507 08:12:09.576545 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 764) took 4241 ms. Trace:
I20240507 08:12:09.576697 6513 rpcz_store.cc:276] 0507 08:12:05.334626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.336122 (+ 1496us) service_pool.cc:225] Handling call
0507 08:12:09.576407 (+4240285us) ts_tablet_manager.cc:1120] Deleting tablet 055a30bfc61b4ff2be9f2418ca59b030
0507 08:12:09.576418 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.576507 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4240136}
I20240507 08:12:09.578032 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 765) took 4243 ms. Trace:
I20240507 08:12:09.578227 6513 rpcz_store.cc:276] 0507 08:12:05.334747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.338883 (+ 4136us) service_pool.cc:225] Handling call
0507 08:12:09.577890 (+4239007us) ts_tablet_manager.cc:1120] Deleting tablet 4ad95383487e458ca162038f6940271b
0507 08:12:09.577901 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.577988 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4238851}
I20240507 08:12:09.578842 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 769) took 4163 ms. Trace:
I20240507 08:12:09.578998 6513 rpcz_store.cc:276] 0507 08:12:05.415779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.423536 (+ 7757us) service_pool.cc:225] Handling call
0507 08:12:09.578672 (+4155136us) ts_tablet_manager.cc:1120] Deleting tablet 93c68ddcfa7c40c180b6b12fabe165d6
0507 08:12:09.578695 (+ 23us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.578796 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4149607}
I20240507 08:12:09.579556 6513 tablet_replica.cc:330] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:09.579846 6513 raft_consensus.cc:2229] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:09.580078 6513 raft_consensus.cc:2258] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.580344 6880 raft_consensus.cc:482] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.580492 6880 raft_consensus.cc:504] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.581204 6880 leader_election.cc:288] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:09.581904 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.582438 5378 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.558727 5843 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:09.583266 5843 ts_tablet_manager.cc:1367] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.119s user 0.002s sys 0.002s
I20240507 08:12:09.584321 5839 tablet_bootstrap.cc:654] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.584170 5843 raft_consensus.cc:348] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.584422 5843 raft_consensus.cc:374] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.584494 5843 raft_consensus.cc:729] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.584867 5843 consensus_queue.cc:260] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.586321 5843 ts_tablet_manager.cc:1398] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:09.587383 5843 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.588577 6886 raft_consensus.cc:482] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.588750 6886 raft_consensus.cc:504] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.589480 6886 leader_election.cc:288] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.590648 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.591786 5584 raft_consensus.cc:3037] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.593734 5839 tablet_bootstrap.cc:492] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.593837 5839 ts_tablet_manager.cc:1367] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.002s
I20240507 08:12:09.595414 5839 raft_consensus.cc:348] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.595573 5839 raft_consensus.cc:374] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.595621 5839 raft_consensus.cc:729] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.595799 5839 consensus_queue.cc:260] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.596235 5839 ts_tablet_manager.cc:1398] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.596529 5839 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.597975 5583 raft_consensus.cc:3037] T 81a1a42a22704974b1ef6cbd17d91854 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.559778 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.604082 5442 raft_consensus.cc:2454] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:09.574467 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b6ba3397e33b4cc7805a25e7535a383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ba3397e33b4cc7805a25e7535a383a already in progress: creating tablet
W20240507 08:12:09.604681 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bfa2139385d34c37a7ac14d78480a779 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa2139385d34c37a7ac14d78480a779 already in progress: creating tablet
I20240507 08:12:09.604902 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c48da19c9b9e4207b267137d240adc5b (table default.Table_71fb81b62c8746bf9376dd00c4250d3a [id=d720bba1d3c34621806158a229c9bf8b]) successfully deleted
W20240507 08:12:09.605084 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2bb648fe98e74248b5cd65603de58a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb648fe98e74248b5cd65603de58a02 already in progress: creating tablet
W20240507 08:12:09.605409 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 055a30bfc61b4ff2be9f2418ca59b030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 055a30bfc61b4ff2be9f2418ca59b030 already in progress: creating tablet
W20240507 08:12:09.605684 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4ad95383487e458ca162038f6940271b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ad95383487e458ca162038f6940271b already in progress: creating tablet
W20240507 08:12:09.605899 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 93c68ddcfa7c40c180b6b12fabe165d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c68ddcfa7c40c180b6b12fabe165d6 already in progress: creating tablet
I20240507 08:12:09.564338 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "975f22194aff40609a3cfa63b9b3aca2" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.606846 5444 raft_consensus.cc:3037] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.608968 7007 raft_consensus.cc:482] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c202eef7f3466fa3e703644b40879d)
I20240507 08:12:09.609123 7007 raft_consensus.cc:504] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:12:09.573686 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.609694 5445 raft_consensus.cc:2454] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:09.615006 5444 raft_consensus.cc:2454] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.619568 5843 tablet_bootstrap.cc:654] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.621884 7007 leader_election.cc:288] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459), 94c202eef7f3466fa3e703644b40879d (127.4.246.252:45297)
I20240507 08:12:09.622800 5274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" candidate_term: 2 candidate_status { last_received { term: 1 index: 1181 } } ignore_live_leader: false dest_uuid: "94c202eef7f3466fa3e703644b40879d" is_pre_election: true
I20240507 08:12:09.626320 5839 tablet_bootstrap.cc:654] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:09.626310 5108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111) [suppressed 7 similar messages]
W20240507 08:12:09.627017 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet be1b36ca5b6041d780e851f137d3a4c9 (table default.Table_c649ec5b00df4a4e851f765549a0d9f8 [id=6325b34aec0b45d981c7a0825903409a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet be1b36ca5b6041d780e851f137d3a4c9 consensus state has no leader
W20240507 08:12:09.630074 5108 leader_election.cc:334] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111)
I20240507 08:12:09.630440 5108 leader_election.cc:302] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a91047a8a5414eff9c06ca9ca2b548e8; no voters: 94c202eef7f3466fa3e703644b40879d, a39de2a69ed4492aab4d7a232b09c1ae
I20240507 08:12:09.630827 7007 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a91047a8a5414eff9c06ca9ca2b548e8 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240507 08:12:09.632045 5839 tablet_bootstrap.cc:492] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.632193 5839 ts_tablet_manager.cc:1367] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.001s
I20240507 08:12:09.632922 5839 raft_consensus.cc:348] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.633098 5839 raft_consensus.cc:374] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.633152 5839 raft_consensus.cc:729] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.633373 5839 consensus_queue.cc:260] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.634598 5839 ts_tablet_manager.cc:1398] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:09.635447 5839 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.636251 5843 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:09.636408 5843 ts_tablet_manager.cc:1367] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.002s
I20240507 08:12:09.636802 5839 tablet_bootstrap.cc:654] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.637290 5843 raft_consensus.cc:348] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.637487 5843 raft_consensus.cc:374] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.637565 5843 raft_consensus.cc:729] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.637835 5843 consensus_queue.cc:260] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.638686 5843 ts_tablet_manager.cc:1398] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:09.639263 5843 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.640755 5843 tablet_bootstrap.cc:654] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.645428 5839 tablet_bootstrap.cc:492] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.645555 5839 ts_tablet_manager.cc:1367] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20240507 08:12:09.646201 5839 raft_consensus.cc:348] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.646373 5839 raft_consensus.cc:374] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.646428 5839 raft_consensus.cc:729] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.646616 5839 consensus_queue.cc:260] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.647629 5843 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:09.647783 5843 ts_tablet_manager.cc:1367] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.004s
I20240507 08:12:09.647763 5839 ts_tablet_manager.cc:1398] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.648666 5839 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.649971 5839 tablet_bootstrap.cc:654] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.653023 6886 raft_consensus.cc:482] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.653182 6886 raft_consensus.cc:504] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.654137 6886 leader_election.cc:288] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:09.655424 5520 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:09.658185 5839 tablet_bootstrap.cc:492] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.658304 5839 ts_tablet_manager.cc:1367] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240507 08:12:09.658921 5839 raft_consensus.cc:348] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.659072 5839 raft_consensus.cc:374] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.659111 5839 raft_consensus.cc:729] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.659540 5839 consensus_queue.cc:260] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.660001 5839 ts_tablet_manager.cc:1398] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.661897 5839 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.662290 5843 raft_consensus.cc:348] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.662509 5843 raft_consensus.cc:374] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.662571 5843 raft_consensus.cc:729] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.582991 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.586347 5840 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.568663 6912 leader_election.cc:288] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.664798 5840 ts_tablet_manager.cc:1367] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.091s user 0.002s sys 0.005s
I20240507 08:12:09.645350 5656 leader_election.cc:302] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:09.665021 5377 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.665135 5377 leader_election.cc:302] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.665220 6912 raft_consensus.cc:2781] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.665341 6912 raft_consensus.cc:482] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.665428 6912 raft_consensus.cc:3037] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.665477 6880 raft_consensus.cc:2726] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.665630 5840 raft_consensus.cc:348] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.665886 5840 raft_consensus.cc:374] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.665984 5840 raft_consensus.cc:729] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.666189 5840 consensus_queue.cc:260] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.666663 5840 ts_tablet_manager.cc:1398] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.666931 5840 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.668032 6912 raft_consensus.cc:504] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.668336 5839 tablet_bootstrap.cc:654] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.668699 5840 tablet_bootstrap.cc:654] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.669144 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:09.669250 6912 leader_election.cc:288] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:09.669682 5654 leader_election.cc:341] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.670082 5656 leader_election.cc:302] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.670392 6912 raft_consensus.cc:2781] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.670552 6912 raft_consensus.cc:482] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.670723 6912 raft_consensus.cc:3037] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.673831 6912 raft_consensus.cc:504] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.674855 6912 leader_election.cc:288] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.675233 5656 leader_election.cc:302] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.675881 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5782cfa5eca74c19a2e724a9124597f2" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.675899 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:09.676507 5654 leader_election.cc:341] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.676735 5839 tablet_bootstrap.cc:492] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.676862 5839 ts_tablet_manager.cc:1367] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.008s
I20240507 08:12:09.677091 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45d3fbbe3764261a3a24523a77c6443" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.677321 5718 raft_consensus.cc:2454] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
I20240507 08:12:09.677662 5839 raft_consensus.cc:348] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.677829 5839 raft_consensus.cc:374] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.677881 5839 raft_consensus.cc:729] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.678054 5839 consensus_queue.cc:260] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:09.678189 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.678442 5444 raft_consensus.cc:3037] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.678493 5839 ts_tablet_manager.cc:1398] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:09.678731 5839 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.680467 5839 tablet_bootstrap.cc:654] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.681921 5444 raft_consensus.cc:2454] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.682024 5840 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.682159 5840 ts_tablet_manager.cc:1367] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.005s
I20240507 08:12:09.682883 5840 raft_consensus.cc:348] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.683068 5840 raft_consensus.cc:374] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.683071 5517 leader_election.cc:302] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.683143 5840 raft_consensus.cc:729] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.683328 5840 consensus_queue.cc:260] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.683578 6886 raft_consensus.cc:2726] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.683583 6675 raft_consensus.cc:2781] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.683724 6675 raft_consensus.cc:686] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.683987 6675 consensus_queue.cc:237] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.683996 5840 ts_tablet_manager.cc:1398] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:09.684451 5517 leader_election.cc:302] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.684522 5840 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:09.684800 6886 raft_consensus.cc:2781] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.684866 6886 raft_consensus.cc:482] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.684922 6886 raft_consensus.cc:3037] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.686046 5840 tablet_bootstrap.cc:654] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.687731 6886 raft_consensus.cc:504] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.688445 6886 leader_election.cc:288] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.688792 5843 consensus_queue.cc:260] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.689464 5520 leader_election.cc:341] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:09.690819 6886 raft_consensus.cc:482] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.691002 5579 raft_consensus.cc:3037] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.690969 6886 raft_consensus.cc:504] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:09.692719 5520 leader_election.cc:341] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 87377661d9374cef8ab1a97042b7f1b9
I20240507 08:12:09.692888 6886 leader_election.cc:288] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.693130 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c7c39ec9fec4331afabb39d1483ef57" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:09.693341 5444 raft_consensus.cc:3037] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.694849 5839 tablet_bootstrap.cc:492] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.694993 5839 ts_tablet_manager.cc:1367] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.007s
I20240507 08:12:09.695991 5839 raft_consensus.cc:348] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.696153 5839 raft_consensus.cc:374] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.696198 5839 raft_consensus.cc:729] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.696362 5839 consensus_queue.cc:260] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.696779 5839 ts_tablet_manager.cc:1398] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:09.697400 5839 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:09.699000 5839 tablet_bootstrap.cc:654] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.699090 5843 ts_tablet_manager.cc:1398] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.051s user 0.001s sys 0.002s
I20240507 08:12:09.700552 5444 raft_consensus.cc:2454] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:09.702013 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45d3fbbe3764261a3a24523a77c6443" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:09.702265 5718 raft_consensus.cc:3037] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.702531 5843 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.706264 5718 raft_consensus.cc:2454] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:12:09.706610 6886 raft_consensus.cc:482] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.706795 6886 raft_consensus.cc:504] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.708009 6886 leader_election.cc:288] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.708173 6886 raft_consensus.cc:482] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.708314 6886 raft_consensus.cc:504] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:09.709823 5520 leader_election.cc:341] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:12:09.710247 6886 leader_election.cc:288] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:09.710848 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87377661d9374cef8ab1a97042b7f1b9" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.711447 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.712180 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7947ca2d8324dbb9c909ec50852fed8" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.713048 6513 ts_tablet_manager.cc:1875] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:09.713474 6784 raft_consensus.cc:482] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.713616 6784 raft_consensus.cc:504] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.714259 6784 leader_election.cc:288] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.715020 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.715595 5654 leader_election.cc:341] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.705638 5843 tablet_bootstrap.cc:654] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.725804 5656 leader_election.cc:302] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.726689 6784 raft_consensus.cc:2781] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.726836 6784 raft_consensus.cc:686] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.727123 6784 consensus_queue.cc:237] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.731896 5656 leader_election.cc:302] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.732311 6784 raft_consensus.cc:2781] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.732553 6784 raft_consensus.cc:686] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:09.733042 6784 consensus_queue.cc:237] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.735200 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed2604a090c94c78bd4f7a1dea6f81b7" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.739395 5517 leader_election.cc:302] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.740141 5517 leader_election.cc:302] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.740639 6886 raft_consensus.cc:2781] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.741665 7019 raft_consensus.cc:2726] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.753859 6934 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.753912 6784 raft_consensus.cc:482] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.754057 6934 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.754114 6784 raft_consensus.cc:504] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.754712 6934 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:09.755594 6784 leader_election.cc:288] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.755548 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:09.755936 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:09.756168 5718 raft_consensus.cc:2454] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:12:09.756117 5378 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.756655 5377 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.757150 6934 raft_consensus.cc:2781] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.757248 6934 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.757354 6934 raft_consensus.cc:3037] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.760561 6934 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.761929 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:09.762586 5378 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.762955 6934 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:09.763257 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:09.763480 5718 raft_consensus.cc:3037] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.766207 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.770092 5654 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.773604 5718 raft_consensus.cc:2454] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:09.774118 5377 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.774863 6880 raft_consensus.cc:2781] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:09.775003 6880 raft_consensus.cc:686] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:09.775313 6880 consensus_queue.cc:237] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.777002 5578 raft_consensus.cc:3037] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.793879 5517 leader_election.cc:302] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.794658 5517 leader_election.cc:302] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:09.796298 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.808918 6675 raft_consensus.cc:482] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.809130 6675 raft_consensus.cc:504] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.809978 6675 leader_election.cc:288] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.810858 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.811569 5654 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.814986 7048 raft_consensus.cc:2726] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.819095 7019 raft_consensus.cc:2726] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:09.821177 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:09.821430 5444 raft_consensus.cc:2454] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:09.822432 5840 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:09.827524 6886 raft_consensus.cc:686] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:12:09.828086 6886 consensus_queue.cc:237] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.829807 7019 raft_consensus.cc:482] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.830416 7019 raft_consensus.cc:504] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.831261 5840 ts_tablet_manager.cc:1367] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.147s user 0.012s sys 0.010s
I20240507 08:12:09.832183 5840 raft_consensus.cc:348] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.832401 5840 raft_consensus.cc:374] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.832484 5840 raft_consensus.cc:729] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.831876 7019 leader_election.cc:288] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.832702 5840 consensus_queue.cc:260] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.833225 5840 ts_tablet_manager.cc:1398] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:09.833528 5840 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:09.847988 5520 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 4716c1285e91432a92b25ab13ada2779
I20240507 08:12:09.847983 7051 tablet.cc:1722] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:09.852299 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet aa094773f3bf418492bcd5b5021fd429 (table default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]). Expected version 1 got 0
I20240507 08:12:09.852540 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:09.855746 5843 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:09.855911 5843 ts_tablet_manager.cc:1367] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.157s user 0.004s sys 0.018s
I20240507 08:12:09.856622 5843 raft_consensus.cc:348] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.856837 5843 raft_consensus.cc:374] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.856920 5843 raft_consensus.cc:729] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.857149 5843 consensus_queue.cc:260] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.851588 5686 tablet_service.cc:1467] Processing CreateTablet for tablet 00b39b474db642efb820338770936949 (DEFAULT_TABLE table=default.Table_d0344e2cea8344ddb56b67b65fdc2c57 [id=982b72cbb1b84f88ba07b78e65381048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.860064 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b39b474db642efb820338770936949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.861455 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 00b39b474db642efb820338770936949 (DEFAULT_TABLE table=default.Table_d0344e2cea8344ddb56b67b65fdc2c57 [id=982b72cbb1b84f88ba07b78e65381048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.861610 5840 tablet_bootstrap.cc:654] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.861986 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b39b474db642efb820338770936949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.852320 6784 raft_consensus.cc:482] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.869868 6784 raft_consensus.cc:504] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.870952 5696 tablet_service.cc:1467] Processing CreateTablet for tablet c245e220d07d4a3f9c98e0c38488f08a (DEFAULT_TABLE table=default.Table_25e1a176f4084371aab7be282c6e6f6d [id=635ad449e2f3496da8a27115fb4bf813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.871407 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245e220d07d4a3f9c98e0c38488f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.871780 5412 tablet_service.cc:1467] Processing CreateTablet for tablet c245e220d07d4a3f9c98e0c38488f08a (DEFAULT_TABLE table=default.Table_25e1a176f4084371aab7be282c6e6f6d [id=635ad449e2f3496da8a27115fb4bf813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.872267 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245e220d07d4a3f9c98e0c38488f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.872653 5843 ts_tablet_manager.cc:1398] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240507 08:12:09.873116 5843 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:09.874763 5843 tablet_bootstrap.cc:654] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:09.876259 6784 leader_election.cc:288] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.876168 7055 tablet.cc:1722] T 169626f9a0bc4c6f9d8be11c0a802816 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:09.876873 7019 raft_consensus.cc:482] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.876987 7019 raft_consensus.cc:504] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.877125 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 06e60e00d8474ddb9549a1b6fa209091 (DEFAULT_TABLE table=default.Table_e81e95a06efd420bbaec15335ed3e556 [id=910e8e453cd3439aa1e1998c047a056d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.877604 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e60e00d8474ddb9549a1b6fa209091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.877826 7019 leader_election.cc:288] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.878603 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 06e60e00d8474ddb9549a1b6fa209091 (DEFAULT_TABLE table=default.Table_e81e95a06efd420bbaec15335ed3e556 [id=910e8e453cd3439aa1e1998c047a056d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.878938 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e60e00d8474ddb9549a1b6fa209091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.887749 5579 raft_consensus.cc:3037] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.889757 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 (DEFAULT_TABLE table=default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab [id=f84610c914c4400b95b1d57f7f213df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.890136 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636e8f123b9540f3afedf02e4df5dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.891223 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 00b39b474db642efb820338770936949 (DEFAULT_TABLE table=default.Table_d0344e2cea8344ddb56b67b65fdc2c57 [id=982b72cbb1b84f88ba07b78e65381048]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.891561 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00b39b474db642efb820338770936949. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.893355 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 83ed63048286415cadae77b750923f0d (DEFAULT_TABLE table=default.Table_775645191cbe4f3aafd1b0ee37db5591 [id=5a0d7b107b14407c9daa0af2163e6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.895198 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ed63048286415cadae77b750923f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.898501 5555 tablet_service.cc:1467] Processing CreateTablet for tablet c245e220d07d4a3f9c98e0c38488f08a (DEFAULT_TABLE table=default.Table_25e1a176f4084371aab7be282c6e6f6d [id=635ad449e2f3496da8a27115fb4bf813]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.898856 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c245e220d07d4a3f9c98e0c38488f08a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.908341 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 (DEFAULT_TABLE table=default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab [id=f84610c914c4400b95b1d57f7f213df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.908772 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636e8f123b9540f3afedf02e4df5dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.910390 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 (DEFAULT_TABLE table=default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab [id=f84610c914c4400b95b1d57f7f213df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.910843 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 636e8f123b9540f3afedf02e4df5dfb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.913659 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.915850 5654 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:09.852458 5656 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.916527 5656 leader_election.cc:302] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:09.917709 5520 leader_election.cc:341] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: dfa14901cdf14b72bea365b04ad521bd
I20240507 08:12:09.917773 6784 raft_consensus.cc:2726] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:09.917958 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbfbcafc930c496ca7c8f008c934e26d consensus state has no leader
W20240507 08:12:09.918565 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf8fe5157dd9491cac7e0237c3d39ec1 consensus state has no leader
I20240507 08:12:09.919411 5841 ts_tablet_manager.cc:1888] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:09.919682 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 83ed63048286415cadae77b750923f0d (DEFAULT_TABLE table=default.Table_775645191cbe4f3aafd1b0ee37db5591 [id=5a0d7b107b14407c9daa0af2163e6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.920076 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ed63048286415cadae77b750923f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.920279 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 06e60e00d8474ddb9549a1b6fa209091 (DEFAULT_TABLE table=default.Table_e81e95a06efd420bbaec15335ed3e556 [id=910e8e453cd3439aa1e1998c047a056d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:09.919788 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 67324us
I20240507 08:12:09.923705 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3b373f17dc40efa05dd024318393c5 (DEFAULT_TABLE table=default.Table_4892af21ff1c43deb37b30758ce5ec7c [id=4dcbf358e5b0455a9398dc166480b56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.924453 5656 leader_election.cc:302] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:09.923936 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3b373f17dc40efa05dd024318393c5 (DEFAULT_TABLE table=default.Table_4892af21ff1c43deb37b30758ce5ec7c [id=4dcbf358e5b0455a9398dc166480b56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.925496 5562 tablet_service.cc:1467] Processing CreateTablet for tablet dba4967c141f4078b6f24d4b71165906 (DEFAULT_TABLE table=default.Table_605d0c8dfbc846a69225b3abf59b892e [id=33eb55046f79435b8a03c65816ad1ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.926280 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3b373f17dc40efa05dd024318393c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.926839 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3b373f17dc40efa05dd024318393c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.923522 5686 tablet_service.cc:1467] Processing CreateTablet for tablet dba4967c141f4078b6f24d4b71165906 (DEFAULT_TABLE table=default.Table_605d0c8dfbc846a69225b3abf59b892e [id=33eb55046f79435b8a03c65816ad1ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.930404 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba4967c141f4078b6f24d4b71165906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.933338 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06e60e00d8474ddb9549a1b6fa209091. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.934604 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba4967c141f4078b6f24d4b71165906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.937361 6784 raft_consensus.cc:2781] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:09.937533 6784 raft_consensus.cc:482] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:09.937726 6784 raft_consensus.cc:3037] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:09.940821 6784 raft_consensus.cc:504] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.941995 6784 leader_election.cc:288] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.942548 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:09.943346 5654 leader_election.cc:341] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.945788 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 7467a94ebb964d1e8bc298eed1ab4375 (DEFAULT_TABLE table=default.Table_d033de6ad84349dab6747ef863ec4355 [id=be51b8d96cc34c1492485a4e1fbba368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.946310 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7467a94ebb964d1e8bc298eed1ab4375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.946555 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 7467a94ebb964d1e8bc298eed1ab4375 (DEFAULT_TABLE table=default.Table_d033de6ad84349dab6747ef863ec4355 [id=be51b8d96cc34c1492485a4e1fbba368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.947083 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7467a94ebb964d1e8bc298eed1ab4375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.955969 5839 tablet_bootstrap.cc:492] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:09.956382 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 567846c488614142b8d29480a2730260 (DEFAULT_TABLE table=default.Table_e3aa977674844b06a78cd9c7a4e183a6 [id=7a9fd62aa40442df9bf098a061f28b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.956674 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 567846c488614142b8d29480a2730260 (DEFAULT_TABLE table=default.Table_e3aa977674844b06a78cd9c7a4e183a6 [id=7a9fd62aa40442df9bf098a061f28b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.956854 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567846c488614142b8d29480a2730260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.957003 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567846c488614142b8d29480a2730260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.963433 5839 ts_tablet_manager.cc:1367] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.266s user 0.072s sys 0.052s
I20240507 08:12:09.964280 5839 raft_consensus.cc:348] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.964533 5839 raft_consensus.cc:374] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:09.964597 5839 raft_consensus.cc:729] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:09.964840 6784 raft_consensus.cc:482] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:09.965065 6784 raft_consensus.cc:504] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.965899 6784 leader_election.cc:288] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:09.968842 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc (DEFAULT_TABLE table=default.Table_93d4d0c49c4b4bf1b92f816195fe7068 [id=548a65cb5188403cb101c15920ed4613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.969049 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc (DEFAULT_TABLE table=default.Table_93d4d0c49c4b4bf1b92f816195fe7068 [id=548a65cb5188403cb101c15920ed4613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.969326 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fcb6a8839840838bedb777ff1cbbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.969455 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fcb6a8839840838bedb777ff1cbbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.974344 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:09.985546 5654 leader_election.cc:341] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.002780 6508 raft_consensus.cc:482] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.002982 6508 raft_consensus.cc:504] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.003630 6508 leader_election.cc:288] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.007169 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:10.007747 5654 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:09.923866 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 6f3b373f17dc40efa05dd024318393c5 (DEFAULT_TABLE table=default.Table_4892af21ff1c43deb37b30758ce5ec7c [id=4dcbf358e5b0455a9398dc166480b56e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.008630 5841 log.cc:1196] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/79524df2415146ab926425c42068626c
I20240507 08:12:10.008827 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f3b373f17dc40efa05dd024318393c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.009193 5841 ts_tablet_manager.cc:1909] T 79524df2415146ab926425c42068626c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.009635 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 752) took 5250 ms. Trace:
I20240507 08:12:10.009747 5841 rpcz_store.cc:276] 0507 08:12:04.759169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.782375 (+ 23206us) service_pool.cc:225] Handling call
0507 08:12:09.516616 (+4734241us) ts_tablet_manager.cc:1120] Deleting tablet 79524df2415146ab926425c42068626c
0507 08:12:09.516627 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.643569 (+126942us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:09.919263 (+275694us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.009560 (+ 90297us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4733916}
I20240507 08:12:10.010325 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 79524df2415146ab926425c42068626c (table default.Table_16b866688308450cb4d4f81ccafa1afb [id=35093b3de3164b9c809aae8b5da3db54]) successfully deleted
I20240507 08:12:10.010782 5841 tablet_replica.cc:330] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.011008 5841 raft_consensus.cc:2229] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.011214 5841 raft_consensus.cc:2258] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:09.935475 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 83ed63048286415cadae77b750923f0d (DEFAULT_TABLE table=default.Table_775645191cbe4f3aafd1b0ee37db5591 [id=5a0d7b107b14407c9daa0af2163e6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.014135 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83ed63048286415cadae77b750923f0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:09.940441 5412 tablet_service.cc:1467] Processing CreateTablet for tablet dba4967c141f4078b6f24d4b71165906 (DEFAULT_TABLE table=default.Table_605d0c8dfbc846a69225b3abf59b892e [id=33eb55046f79435b8a03c65816ad1ddf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.944723 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.015960 5444 raft_consensus.cc:2454] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.016047 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dba4967c141f4078b6f24d4b71165906. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.021148 5839 consensus_queue.cc:260] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.954053 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 7467a94ebb964d1e8bc298eed1ab4375 (DEFAULT_TABLE table=default.Table_d033de6ad84349dab6747ef863ec4355 [id=be51b8d96cc34c1492485a4e1fbba368]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.021991 5839 ts_tablet_manager.cc:1398] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.058s user 0.001s sys 0.001s
I20240507 08:12:10.022120 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7467a94ebb964d1e8bc298eed1ab4375. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.022464 5839 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:10.024125 7044 raft_consensus.cc:482] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.024255 7044 raft_consensus.cc:504] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:09.966020 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 567846c488614142b8d29480a2730260 (DEFAULT_TABLE table=default.Table_e3aa977674844b06a78cd9c7a4e183a6 [id=7a9fd62aa40442df9bf098a061f28b35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.024821 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 567846c488614142b8d29480a2730260. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.025087 7044 leader_election.cc:288] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:10.026275 5520 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e555a89d340448bc852f2441ca850213
I20240507 08:12:09.980841 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc (DEFAULT_TABLE table=default.Table_93d4d0c49c4b4bf1b92f816195fe7068 [id=548a65cb5188403cb101c15920ed4613]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:09.983912 6880 raft_consensus.cc:482] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.029908 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71fcb6a8839840838bedb777ff1cbbcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.029757 6880 raft_consensus.cc:504] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.030537 6880 leader_election.cc:288] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.002633 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.030771 5442 raft_consensus.cc:2454] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.004256 5440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.031240 5440 raft_consensus.cc:3037] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.035300 5440 raft_consensus.cc:2454] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.006328 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.036986 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:10.037715 5378 leader_election.cc:341] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.038244 6508 raft_consensus.cc:482] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.038417 6508 raft_consensus.cc:504] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.039140 6508 leader_election.cc:288] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.043339 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.046036 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_28dc4a4ae86d46e79217cc266d671289" } modify_external_catalogs: true
I20240507 08:12:10.046305 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_28dc4a4ae86d46e79217cc266d671289" } modify_external_catalogs: true
I20240507 08:12:10.048105 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:10.046887 6886 raft_consensus.cc:482] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.048418 6886 raft_consensus.cc:504] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.048746 5263 catalog_manager.cc:5484] T 371ba277be774d32912d1174bec68b51 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:10.048945 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a34af5c61e5649e5abc9c88235891674"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.049371 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (table default.Table_87bb488658cd4819a51ff1ff0993ceab [id=799497aabd9b4f17bc42b655e947b3ed]). Expected version 3 got 0
W20240507 08:12:10.049523 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a34af5c61e5649e5abc9c88235891674 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:10.050567 5654 leader_election.cc:341] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.050693 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a425a90f0449bcaaacb4c3f653d26c" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.050776 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9e923448a06e4222b2ab2e13dcff9601"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.051252 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e923448a06e4222b2ab2e13dcff9601 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.053897 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:10.054531 5377 leader_election.cc:341] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.054538 6675 consensus_queue.cc:1035] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.054689 5377 leader_election.cc:302] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.055124 6880 raft_consensus.cc:2726] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.055275 5656 leader_election.cc:302] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.055581 6675 raft_consensus.cc:2781] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.055677 6675 raft_consensus.cc:482] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.055773 6675 raft_consensus.cc:3037] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.058661 6675 raft_consensus.cc:504] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.059281 6675 leader_election.cc:288] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.061062 6513 ts_tablet_manager.cc:1888] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.061187 6513 log.cc:1196] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d451d03349794726a71c255a7f177436
I20240507 08:12:10.061492 6513 ts_tablet_manager.cc:1909] T d451d03349794726a71c255a7f177436 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.062366 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 771) took 4593 ms. Trace:
I20240507 08:12:10.062758 6886 leader_election.cc:288] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.062786 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:10.069301 5841 ts_tablet_manager.cc:1875] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.071125 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.071537 5839 tablet_bootstrap.cc:654] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.071511 5445 raft_consensus.cc:2454] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:10.063323 5654 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.068670 5656 leader_election.cc:302] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.084060 6784 raft_consensus.cc:2781] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.084213 6784 raft_consensus.cc:482] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.084421 6784 raft_consensus.cc:3037] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.086979 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 06898414c5494cd380e0c932f4589014
I20240507 08:12:10.087234 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1ab94b1a09dd4969b928b017229ce4bf
I20240507 08:12:10.087616 5255 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:44738 (request call id 1135) took 1217 ms. Trace:
I20240507 08:12:10.087610 6784 raft_consensus.cc:504] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.087724 5255 rpcz_store.cc:276] 0507 08:12:08.869638 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.948853 (+ 79215us) service_pool.cc:225] Handling call
0507 08:12:08.949295 (+ 442us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0507 08:12:08.949446 (+ 151us) catalog_manager.cc:2749] Modifying in-memory table state
0507 08:12:08.949522 (+ 76us) catalog_manager.cc:2758] Locking tablets
0507 08:12:09.849414 (+899892us) catalog_manager.cc:2772] Removing table and tablets from system table
0507 08:12:10.086742 (+237328us) catalog_manager.cc:2790] Removing table from by-name map
0507 08:12:10.086852 (+ 110us) catalog_manager.cc:2806] Committing in-memory state
0507 08:12:10.086920 (+ 68us) catalog_manager.cc:2810] Aborting table tasks
0507 08:12:10.087550 (+ 630us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:09.994441 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:09.994606 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:09.994613 (+ 7us) write_op.cc:435] Acquired schema lock
0507 08:12:09.994615 (+ 2us) tablet.cc:660] Decoding operations
0507 08:12:09.994762 (+ 147us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:09.994777 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:09.994782 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:09.994928 (+ 146us) tablet.cc:699] Row locks acquired
0507 08:12:09.994931 (+ 3us) write_op.cc:260] PREPARE: finished
0507 08:12:09.994973 (+ 42us) write_op.cc:270] Start()
0507 08:12:09.995008 (+ 35us) write_op.cc:276] Timestamp: P: 1715069529994932 usec, L: 0
0507 08:12:09.995012 (+ 4us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.072320 (+ 77308us) log.cc:842] Serialized 762 byte log entry
0507 08:12:10.073555 (+ 1235us) spinlock_profiling.cc:243] Waited 77 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 000000000095519a 00007f3bc4c1f162 00007f3bc5f87e32 00007f3bc5f86959 00007f3bc5f8e86b 00007f3bc5f8e6e8 0000000000a60bcc 00007f3bb9420b74 00007f3bb942419b 00007f3bb9424018 0000000000a60bcc
0507 08:12:10.086135 (+ 12580us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.086256 (+ 121us) write_op.cc:301] APPLY: starting
0507 08:12:10.086270 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.086313 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.086316 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.086389 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.086424 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0507 08:12:10.086432 (+ 8us) write_op.cc:312] APPLY: finished
0507 08:12:10.086501 (+ 69us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.086569 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.086590 (+ 21us) write_op.cc:454] Released schema lock
0507 08:12:10.086598 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.086653 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.086657 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":448,"apply.run_wall_time_us":448,"num_ops":2,"prepare.queue_time_us":144065,"prepare.run_cpu_time_us":1503,"prepare.run_wall_time_us":79221,"raft.queue_time_us":3404,"raft.run_cpu_time_us":316,"raft.run_wall_time_us":9032,"replication_time_us":91104,"spinlock_wait_cycles":177090816,"thread_start_us":144808,"threads_started":1}]]}
I20240507 08:12:10.088516 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:10.088658 6784 leader_election.cc:288] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.088806 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:10.089001 5654 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.089112 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 06898414c5494cd380e0c932f4589014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.089303 5656 leader_election.cc:302] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:10.089862 5656 leader_election.cc:341] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: SHUTDOWN
I20240507 08:12:10.090020 5656 leader_election.cc:302] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.090286 6675 raft_consensus.cc:2781] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.090427 6675 raft_consensus.cc:686] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.090745 6675 consensus_queue.cc:237] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.091646 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.091674 6508 raft_consensus.cc:2726] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.092082 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 06898414c5494cd380e0c932f4589014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.093140 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2e44c3f6f96e417b92878d8ba4ff88a7
I20240507 08:12:10.094786 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.095551 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.096367 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.097963 5843 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:10.098163 5843 ts_tablet_manager.cc:1367] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.225s user 0.011s sys 0.012s
I20240507 08:12:10.099152 5843 raft_consensus.cc:348] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.099442 5843 raft_consensus.cc:374] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.099535 5843 raft_consensus.cc:729] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.104384 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0f7492244d9a44b3998beb1d5a959eeb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.104928 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f7492244d9a44b3998beb1d5a959eeb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.106082 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d563809a6ab04242827f32b2b0a6a40f" } new_table_name: "default.Table_dffc373d1b8940d6bdcc3280f92d0743" modify_external_catalogs: true
I20240507 08:12:10.107250 6508 raft_consensus.cc:482] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.107414 6508 raft_consensus.cc:504] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.108027 6508 leader_election.cc:288] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.108068 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c39926d459d14c72884a82bf549db3ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.108493 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c39926d459d14c72884a82bf549db3ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.108728 5656 leader_election.cc:302] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.108712 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:10.109148 5654 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.109450 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 06898414c5494cd380e0c932f4589014 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.110278 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.110580 6513 rpcz_store.cc:276] 0507 08:12:05.468864 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.470038 (+ 1174us) service_pool.cc:225] Handling call
0507 08:12:09.579488 (+4109450us) ts_tablet_manager.cc:1120] Deleting tablet d451d03349794726a71c255a7f177436
0507 08:12:09.579500 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:09.921090 (+341590us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.060959 (+139869us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.062331 (+ 1372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4109269}
I20240507 08:12:10.111053 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d451d03349794726a71c255a7f177436 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
I20240507 08:12:10.112272 6508 raft_consensus.cc:2781] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.112365 6508 raft_consensus.cc:482] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.112455 6508 raft_consensus.cc:3037] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.115024 6508 raft_consensus.cc:504] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.115954 6508 leader_election.cc:288] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.116573 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_70c8dabdda144b77a238abeacaa20e6e [id=95b44cbcf26f489c8f4d8e07cc615776] alter complete (version 2)
W20240507 08:12:10.117559 5257 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.193:55019 (request call id 136) took 951 ms (client timeout 1000 ms). Trace:
I20240507 08:12:10.117832 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bd6ca1a20df748e097de097d1c2d7be3
W20240507 08:12:10.117980 5517 leader_election.cc:341] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 4716c1285e91432a92b25ab13ada2779
I20240507 08:12:10.118115 5517 leader_election.cc:302] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.118337 5264 catalog_manager.cc:6554] ReplaceTablet: tablet bd6ca1a20df748e097de097d1c2d7be3 deleted and replaced by tablet e241dd3c1af14b278b1632481edd0f30
I20240507 08:12:10.118458 5264 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:44738 (request call id 1147) took 1041 ms. Trace:
W20240507 08:12:10.118502 5517 leader_election.cc:341] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: dfa14901cdf14b72bea365b04ad521bd
I20240507 08:12:10.118602 5517 leader_election.cc:302] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.118552 5264 rpcz_store.cc:276] 0507 08:12:09.076635 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.089723 (+ 13088us) service_pool.cc:225] Handling call
0507 08:12:10.118426 (+1028703us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.078920 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.078999 (+ 79us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.079003 (+ 4us) write_op.cc:435] Acquired schema lock
0507 08:12:10.079397 (+ 394us) tablet.cc:660] Decoding operations
0507 08:12:10.079487 (+ 90us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.079495 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.079499 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:10.079583 (+ 84us) tablet.cc:699] Row locks acquired
0507 08:12:10.079585 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:10.079610 (+ 25us) write_op.cc:270] Start()
0507 08:12:10.079633 (+ 23us) write_op.cc:276] Timestamp: P: 1715069530079563 usec, L: 0
0507 08:12:10.079636 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.079757 (+ 121us) log.cc:842] Serialized 628 byte log entry
0507 08:12:10.116065 (+ 36308us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.116167 (+ 102us) write_op.cc:301] APPLY: starting
0507 08:12:10.116180 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.116222 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.116257 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.116343 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.116403 (+ 60us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:12:10.116409 (+ 6us) write_op.cc:312] APPLY: finished
0507 08:12:10.116466 (+ 57us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.116508 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.116523 (+ 15us) write_op.cc:454] Released schema lock
0507 08:12:10.116528 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.116583 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.116587 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":61,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":465,"num_ops":2,"prepare.queue_time_us":224323,"prepare.run_cpu_time_us":493,"prepare.run_wall_time_us":1164,"replication_time_us":36400}]]}
I20240507 08:12:10.119113 7044 raft_consensus.cc:2726] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.119175 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 772) took 4577 ms. Trace:
I20240507 08:12:10.119292 7044 raft_consensus.cc:2726] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.119261 6513 rpcz_store.cc:276] 0507 08:12:05.542012 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.542081 (+ 69us) service_pool.cc:225] Handling call
0507 08:12:10.119028 (+4576947us) ts_tablet_manager.cc:1120] Deleting tablet 2b13fbddcab5460db69f43e63f490fba
0507 08:12:10.119041 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.119140 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4576725}
I20240507 08:12:10.119657 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d17b6a2f4aa14f2a927fa9ad56a62a5a
I20240507 08:12:10.120285 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.120929 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 with delete_type TABLET_DATA_DELETED (replaced by tablet e241dd3c1af14b278b1632481edd0f30) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.121485 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 with delete_type TABLET_DATA_DELETED (replaced by tablet e241dd3c1af14b278b1632481edd0f30) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.076941 5840 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.121727 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.121927 5840 ts_tablet_manager.cc:1367] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.288s user 0.000s sys 0.026s
I20240507 08:12:10.117497 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:10.122809 5654 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:10.123631 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2b13fbddcab5460db69f43e63f490fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b13fbddcab5460db69f43e63f490fba already in progress: creating tablet
I20240507 08:12:10.123935 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.124126 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet bd6ca1a20df748e097de097d1c2d7be3 with delete_type TABLET_DATA_DELETED (replaced by tablet e241dd3c1af14b278b1632481edd0f30) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:10.117719 5257 rpcz_store.cc:269] 0507 08:12:09.166557 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.166625 (+ 68us) service_pool.cc:225] Handling call
0507 08:12:10.117516 (+950891us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.078141 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.078232 (+ 91us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.078237 (+ 5us) write_op.cc:435] Acquired schema lock
0507 08:12:10.078240 (+ 3us) tablet.cc:660] Decoding operations
0507 08:12:10.078311 (+ 71us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.078320 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.078327 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:12:10.078372 (+ 45us) tablet.cc:699] Row locks acquired
0507 08:12:10.078374 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:10.078398 (+ 24us) write_op.cc:270] Start()
0507 08:12:10.078421 (+ 23us) write_op.cc:276] Timestamp: P: 1715069530078351 usec, L: 0
0507 08:12:10.078424 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.078542 (+ 118us) log.cc:842] Serialized 356 byte log entry
0507 08:12:10.115988 (+ 37446us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.116093 (+ 105us) write_op.cc:301] APPLY: starting
0507 08:12:10.116108 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.116141 (+ 33us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.116165 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.116211 (+ 46us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.116246 (+ 35us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:12:10.116253 (+ 7us) write_op.cc:312] APPLY: finished
0507 08:12:10.116312 (+ 59us) log.cc:842] Serialized 23 byte log entry
0507 08:12:10.116381 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.116394 (+ 13us) write_op.cc:454] Released schema lock
0507 08:12:10.116400 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":39,"apply.run_cpu_time_us":456,"apply.run_wall_time_us":4723,"num_ops":1,"prepare.queue_time_us":224514,"prepare.run_cpu_time_us":460,"prepare.run_wall_time_us":740,"replication_time_us":37545}]]}
I20240507 08:12:10.124794 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.122860 5843 consensus_queue.cc:260] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.125000 5445 raft_consensus.cc:2454] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:10.136214 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1ab94b1a09dd4969b928b017229ce4bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab94b1a09dd4969b928b017229ce4bf already in progress: creating tablet
W20240507 08:12:10.136948 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 06898414c5494cd380e0c932f4589014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06898414c5494cd380e0c932f4589014 already in progress: creating tablet
W20240507 08:12:10.137616 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e44c3f6f96e417b92878d8ba4ff88a7 already in progress: creating tablet
W20240507 08:12:10.138392 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17b6a2f4aa14f2a927fa9ad56a62a5a already in progress: creating tablet
W20240507 08:12:10.139029 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bd6ca1a20df748e097de097d1c2d7be3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6ca1a20df748e097de097d1c2d7be3 already in progress: creating tablet
I20240507 08:12:10.139878 5843 ts_tablet_manager.cc:1398] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240507 08:12:10.140439 5843 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:10.142225 5843 tablet_bootstrap.cc:654] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.143795 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "753019050fcf4dc3b69722c3eb475037" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.143858 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0f33de0d21b1417680053925d89ecf91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.144088 5438 raft_consensus.cc:3037] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:10.144501 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f33de0d21b1417680053925d89ecf91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.147377 5438 raft_consensus.cc:2454] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.148193 5262 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]). Expected version 1 got 0
W20240507 08:12:10.149291 5262 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.194:35513 (request call id 163) took 961 ms (client timeout 1000 ms). Trace:
I20240507 08:12:10.149240 5840 raft_consensus.cc:348] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.149474 5840 raft_consensus.cc:374] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:10.149425 5262 rpcz_store.cc:269] 0507 08:12:09.187408 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.848834 (+661426us) service_pool.cc:225] Handling call
0507 08:12:10.047968 (+199134us) spinlock_profiling.cc:243] Waited 199 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9bf668 00007f3bc39b3786 00007f3bc39b34b5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.149248 (+101280us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":456724736}
I20240507 08:12:10.149858 5438 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f294baf04bf74dfe8500cb8c1452d8b3" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.150018 5438 raft_consensus.cc:3037] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.157831 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.160049 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet aa094773f3bf418492bcd5b5021fd429 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:10.161070 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet c48da19c9b9e4207b267137d240adc5b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.164927 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.165156 5442 raft_consensus.cc:2454] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.166195 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 4ad95383487e458ca162038f6940271b from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:10.166687 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0b9f85cd78748d3afce83b64b3f823b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.166898 5442 raft_consensus.cc:3037] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.168004 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6991af2fca624791af1a67498a53b156"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.168534 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6991af2fca624791af1a67498a53b156 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.169745 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_62410585affa4ee1aab69399909e9ff6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.170109 5442 raft_consensus.cc:2454] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:10.170303 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62410585affa4ee1aab69399909e9ff6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.171598 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.173803 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_87bb488658cd4819a51ff1ff0993ceab" } modify_external_catalogs: true
I20240507 08:12:10.174021 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_87bb488658cd4819a51ff1ff0993ceab" } modify_external_catalogs: true
I20240507 08:12:10.175310 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 8497f8ea3c9a48d281d91c90ea305cb2 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:10.176601 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b43357777b9341f4bbd950ab972819a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.177182 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b43357777b9341f4bbd950ab972819a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.177992 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e667ff30b7ca4e44afc8675ec6981a0e" } new_table_name: "default.Table_44eb65d8c630424989eb6b54fbd21962" modify_external_catalogs: true
I20240507 08:12:10.178483 5840 raft_consensus.cc:729] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.178792 5840 consensus_queue.cc:260] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.179412 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cdd1eb0440bf4b9fbf81d06633d206eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.179736 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:10.179950 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdd1eb0440bf4b9fbf81d06633d206eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.179766 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.180235 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.180452 5718 raft_consensus.cc:2454] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
I20240507 08:12:10.180694 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.180895 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 775) took 4482 ms. Trace:
I20240507 08:12:10.181051 6513 rpcz_store.cc:276] 0507 08:12:05.692620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.692674 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:10.142265 (+4449591us) ts_tablet_manager.cc:1120] Deleting tablet c9e2deaadfe34c2dbdc9819729cb4890
0507 08:12:10.174904 (+ 32639us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.175012 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4449347}
W20240507 08:12:10.181646 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c9e2deaadfe34c2dbdc9819729cb4890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e2deaadfe34c2dbdc9819729cb4890 already in progress: creating tablet
W20240507 08:12:10.183544 5654 consensus_peers.cc:485] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.185499 6513 tablet_replica.cc:330] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.185925 6513 raft_consensus.cc:2229] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.186342 6513 raft_consensus.cc:2258] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.187032 6513 ts_tablet_manager.cc:1875] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.190395 5839 tablet_bootstrap.cc:492] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:10.190531 5839 ts_tablet_manager.cc:1367] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.004s
I20240507 08:12:10.191251 5839 raft_consensus.cc:348] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.191444 5839 raft_consensus.cc:374] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.191493 5839 raft_consensus.cc:729] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:10.192575 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c48da19c9b9e4207b267137d240adc5b because the tablet was not found. No further retry: Not found: Tablet not found: c48da19c9b9e4207b267137d240adc5b
I20240507 08:12:10.198447 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:10.198819 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
I20240507 08:12:10.200124 7088 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:10.200255 7081 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:10.203363 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 47d8ca2d20444d00b3a824770d9dc3f4 (DEFAULT_TABLE table=default.Table_5ff9542c98ae41fe80e979ae729f94cc [id=63a6f81c3a35483da5d65c19a29bc4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.203647 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:10.203768 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d8ca2d20444d00b3a824770d9dc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.204056 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 5782cfa5eca74c19a2e724a9124597f2 (table default.Table_ee5279fc19a046aebae1f622389f59a2 [id=c6a13f9902b0424ca8df2184779a407c]). Expected version 3 got 0
I20240507 08:12:10.204591 5656 leader_election.cc:302] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.205056 6508 raft_consensus.cc:2781] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.205153 6508 raft_consensus.cc:482] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.205237 6508 raft_consensus.cc:3037] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.208307 6508 raft_consensus.cc:504] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.208979 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_601dd7f065e343eba100d1a64f0a72a0" } new_table_name: "default.Table_b0b8b0dd37214a57bc0369f2519d0978" modify_external_catalogs: true
I20240507 08:12:10.209255 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:10.209756 5654 leader_election.cc:341] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.209453 6508 leader_election.cc:288] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.215435 6934 raft_consensus.cc:482] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.215637 6934 raft_consensus.cc:504] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.216493 6934 leader_election.cc:288] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.217367 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.217844 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.218586 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 9ccbe2e368d14b56859d62d1cc742f90 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:10.218912 7055 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:10.219991 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 47d8ca2d20444d00b3a824770d9dc3f4 (DEFAULT_TABLE table=default.Table_5ff9542c98ae41fe80e979ae729f94cc [id=63a6f81c3a35483da5d65c19a29bc4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.219997 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 47d8ca2d20444d00b3a824770d9dc3f4 (DEFAULT_TABLE table=default.Table_5ff9542c98ae41fe80e979ae729f94cc [id=63a6f81c3a35483da5d65c19a29bc4fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.220392 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d8ca2d20444d00b3a824770d9dc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.220446 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d8ca2d20444d00b3a824770d9dc3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.179605 5840 ts_tablet_manager.cc:1398] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.058s user 0.000s sys 0.002s
I20240507 08:12:10.223090 5840 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:10.226235 5377 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.227113 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 (DEFAULT_TABLE table=default.Table_cd9885f1f58049c599b7710b4e1f8977 [id=68bd0d8485c74121a3f338e2b6404584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.227474 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372dd1ea1a4043c9ad5afb1d48dfc133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.229858 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 68981cf4d9e74d41b55c411b4c678dab (DEFAULT_TABLE table=default.Table_f1ba0da75f11411ab9babc78e508f6e8 [id=53eff278109c42259cc676e933380806]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.230195 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68981cf4d9e74d41b55c411b4c678dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.233071 5424 tablet_service.cc:1467] Processing CreateTablet for tablet b2028f0db9104597adde7894ae14870c (DEFAULT_TABLE table=default.Table_2139500f65f9422c8a1662c61b6bb1c5 [id=521a98f62e84451a8a5cd95b0718aa41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.233444 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2028f0db9104597adde7894ae14870c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.237126 5656 leader_election.cc:302] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.237463 6912 raft_consensus.cc:2781] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.237650 6912 raft_consensus.cc:686] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.237730 5656 leader_election.cc:302] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.237947 6912 consensus_queue.cc:237] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.238169 5656 leader_election.cc:302] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.238531 6784 raft_consensus.cc:2781] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.238978 6784 raft_consensus.cc:482] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.239174 6784 raft_consensus.cc:3037] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.239629 6675 raft_consensus.cc:2781] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.240617 5555 tablet_service.cc:1467] Processing CreateTablet for tablet b2028f0db9104597adde7894ae14870c (DEFAULT_TABLE table=default.Table_2139500f65f9422c8a1662c61b6bb1c5 [id=521a98f62e84451a8a5cd95b0718aa41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.240870 5438 raft_consensus.cc:2454] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.241024 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2028f0db9104597adde7894ae14870c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:10.241240 5378 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.241361 5378 leader_election.cc:302] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.241902 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 68981cf4d9e74d41b55c411b4c678dab (DEFAULT_TABLE table=default.Table_f1ba0da75f11411ab9babc78e508f6e8 [id=53eff278109c42259cc676e933380806]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.242204 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68981cf4d9e74d41b55c411b4c678dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.242456 5841 ts_tablet_manager.cc:1888] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.242565 5841 log.cc:1196] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/302fbf120e9d472ba6bdf9e6cbdf6dbd
I20240507 08:12:10.242681 6675 raft_consensus.cc:686] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.242900 5841 ts_tablet_manager.cc:1909] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.242933 6675 consensus_queue.cc:237] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.243598 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 755) took 5440 ms. Trace:
I20240507 08:12:10.243686 5841 rpcz_store.cc:276] 0507 08:12:04.803044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.813290 (+ 10246us) service_pool.cc:225] Handling call
0507 08:12:10.010697 (+5197407us) ts_tablet_manager.cc:1120] Deleting tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd
0507 08:12:10.010709 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.081496 (+ 70787us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.242362 (+160866us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.243563 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5197156}
I20240507 08:12:10.244452 5840 tablet_bootstrap.cc:654] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.245802 5547 tablet_service.cc:1467] Processing CreateTablet for tablet df515d25403645f29d186c611907b680 (DEFAULT_TABLE table=default.Table_c05cf821365c4b04a6992a5daf8cc22f [id=d8e4c8a0252745999746370ac26761b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.246017 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 122867a66b58468f86feb5ce4282555e (DEFAULT_TABLE table=default.Table_2365998a0bb04c22bea71d6a205a57cf [id=7f78686b0db5476e9e56152483fe0e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.246150 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df515d25403645f29d186c611907b680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.246340 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122867a66b58468f86feb5ce4282555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.246601 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb529e041148268d1deb653ce983c7 (DEFAULT_TABLE table=default.Table_d9cadcb91f984f24ac8f220b42514007 [id=9958b62f46684aa198cc0bbe6b880a6b]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:10.246680 5654 consensus_peers.cc:485] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.246891 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb529e041148268d1deb653ce983c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.249090 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba515d291b4c77ab4d37e26450d97d (DEFAULT_TABLE table=default.Table_45578b6c891e4ababea94a9331fa91b7 [id=4264dda78665458e91463c3632817861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.249397 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba515d291b4c77ab4d37e26450d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.249780 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 80bfaa8d097646e3bcce7022f314f4fa (DEFAULT_TABLE table=default.Table_c4186447e2f943c5b1eedcb89625e756 [id=fe71f7d3038e4f248f0122c1aefd4770]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.249833 6784 raft_consensus.cc:504] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.250097 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bfaa8d097646e3bcce7022f314f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.250861 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:10.251338 5654 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.251394 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 758) took 5274 ms. Trace:
I20240507 08:12:10.251497 5841 rpcz_store.cc:276] 0507 08:12:04.976649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.976700 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:10.251239 (+5274539us) ts_tablet_manager.cc:1120] Deleting tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5
0507 08:12:10.251256 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.251355 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5274328}
I20240507 08:12:10.251981 6880 raft_consensus.cc:2726] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.253806 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 6549db5bfffe45d98930a2a615db6948 (DEFAULT_TABLE table=default.Table_f932b65c022243ba8acd49cc96e5dff1 [id=d051bd1329b041348df36aa4639f9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.254065 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 4452dbc20b47409f88d8859143f1ac17 (DEFAULT_TABLE table=default.Table_a79ee8acf6384968a6fffcbb9284c2eb [id=802677ef492642f28a0182514533f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.254356 6513 ts_tablet_manager.cc:1888] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:10.254479 6513 log.cc:1196] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/077ccc3946b34008bea7b80d8aee2662
I20240507 08:12:10.245811 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 462bec9653494567b45f2c3a0943f921 (DEFAULT_TABLE table=default.Table_d553689c30b44cef8f22ed8aa85e9873 [id=1bb6bd61322a4c2caabdfe5a296ff697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.254143 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6549db5bfffe45d98930a2a615db6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.255206 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 761) took 5266 ms. Trace:
I20240507 08:12:10.255297 5841 rpcz_store.cc:276] 0507 08:12:04.989114 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.989188 (+ 74us) service_pool.cc:225] Handling call
0507 08:12:10.254945 (+5265757us) ts_tablet_manager.cc:1120] Deleting tablet ed0018356e254b82995061ef7290556d
0507 08:12:10.254955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.255088 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5265545}
I20240507 08:12:10.255838 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 (DEFAULT_TABLE table=default.Table_cd9885f1f58049c599b7710b4e1f8977 [id=68bd0d8485c74121a3f338e2b6404584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.256201 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 80bfaa8d097646e3bcce7022f314f4fa (DEFAULT_TABLE table=default.Table_c4186447e2f943c5b1eedcb89625e756 [id=fe71f7d3038e4f248f0122c1aefd4770]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.256603 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bfaa8d097646e3bcce7022f314f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.256582 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (table default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]) successfully deleted
W20240507 08:12:10.256856 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ba6c1e817bc43dab5dbc7bf4b5786b5 already in progress: creating tablet
I20240507 08:12:10.257061 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372dd1ea1a4043c9ad5afb1d48dfc133. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:10.257074 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ed0018356e254b82995061ef7290556d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ed0018356e254b82995061ef7290556d already in progress: creating tablet
I20240507 08:12:10.257454 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 80bfaa8d097646e3bcce7022f314f4fa (DEFAULT_TABLE table=default.Table_c4186447e2f943c5b1eedcb89625e756 [id=fe71f7d3038e4f248f0122c1aefd4770]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.257865 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80bfaa8d097646e3bcce7022f314f4fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.257866 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 4452dbc20b47409f88d8859143f1ac17 (DEFAULT_TABLE table=default.Table_a79ee8acf6384968a6fffcbb9284c2eb [id=802677ef492642f28a0182514533f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.257897 5416 tablet_service.cc:1467] Processing CreateTablet for tablet df515d25403645f29d186c611907b680 (DEFAULT_TABLE table=default.Table_c05cf821365c4b04a6992a5daf8cc22f [id=d8e4c8a0252745999746370ac26761b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.258531 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb529e041148268d1deb653ce983c7 (DEFAULT_TABLE table=default.Table_d9cadcb91f984f24ac8f220b42514007 [id=9958b62f46684aa198cc0bbe6b880a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.258620 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 759) took 5269 ms. Trace:
I20240507 08:12:10.258704 5841 rpcz_store.cc:276] 0507 08:12:04.988804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.989188 (+ 384us) service_pool.cc:225] Handling call
0507 08:12:10.257971 (+5268783us) ts_tablet_manager.cc:1120] Deleting tablet 96ef77ef4f474107bf31395f5f30c12f
0507 08:12:10.257980 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.258577 (+ 597us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5268462}
I20240507 08:12:10.259034 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 462bec9653494567b45f2c3a0943f921 (DEFAULT_TABLE table=default.Table_d553689c30b44cef8f22ed8aa85e9873 [id=1bb6bd61322a4c2caabdfe5a296ff697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.259090 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462bec9653494567b45f2c3a0943f921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.259383 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462bec9653494567b45f2c3a0943f921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.261363 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 760) took 5272 ms. Trace:
I20240507 08:12:10.261453 5841 rpcz_store.cc:276] 0507 08:12:04.988911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.989271 (+ 360us) service_pool.cc:225] Handling call
0507 08:12:10.261223 (+5271952us) ts_tablet_manager.cc:1120] Deleting tablet 8a397c40278141b7bb80ea5fcdbb125b
0507 08:12:10.261233 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.261322 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5271664}
I20240507 08:12:10.261698 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452dbc20b47409f88d8859143f1ac17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.261826 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 462bec9653494567b45f2c3a0943f921 (DEFAULT_TABLE table=default.Table_d553689c30b44cef8f22ed8aa85e9873 [id=1bb6bd61322a4c2caabdfe5a296ff697]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.262109 6513 ts_tablet_manager.cc:1909] T 077ccc3946b34008bea7b80d8aee2662 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.262156 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 122867a66b58468f86feb5ce4282555e (DEFAULT_TABLE table=default.Table_2365998a0bb04c22bea71d6a205a57cf [id=7f78686b0db5476e9e56152483fe0e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.262331 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 122867a66b58468f86feb5ce4282555e (DEFAULT_TABLE table=default.Table_2365998a0bb04c22bea71d6a205a57cf [id=7f78686b0db5476e9e56152483fe0e07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.262521 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122867a66b58468f86feb5ce4282555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.262848 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba515d291b4c77ab4d37e26450d97d (DEFAULT_TABLE table=default.Table_45578b6c891e4ababea94a9331fa91b7 [id=4264dda78665458e91463c3632817861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.262946 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 762) took 5269 ms. Trace:
I20240507 08:12:10.263034 5841 rpcz_store.cc:276] 0507 08:12:04.993263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:04.993318 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:10.262800 (+5269482us) ts_tablet_manager.cc:1120] Deleting tablet 4e3ea4658bb946439e4a50962444b3e2
0507 08:12:10.262808 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.262908 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5269294}
I20240507 08:12:10.263273 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 6549db5bfffe45d98930a2a615db6948 (DEFAULT_TABLE table=default.Table_f932b65c022243ba8acd49cc96e5dff1 [id=d051bd1329b041348df36aa4639f9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.265074 7088 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:10.258558 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df515d25403645f29d186c611907b680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.265866 5841 tablet_replica.cc:330] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.266073 5841 raft_consensus.cc:2229] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.266060 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 93ba515d291b4c77ab4d37e26450d97d (DEFAULT_TABLE table=default.Table_45578b6c891e4ababea94a9331fa91b7 [id=4264dda78665458e91463c3632817861]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.266261 5841 raft_consensus.cc:2258] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.266404 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba515d291b4c77ab4d37e26450d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.267577 5841 ts_tablet_manager.cc:1875] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.267721 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6549db5bfffe45d98930a2a615db6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.269713 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 122867a66b58468f86feb5ce4282555e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.273764 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb529e041148268d1deb653ce983c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.274165 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452dbc20b47409f88d8859143f1ac17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.274889 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 462bec9653494567b45f2c3a0943f921. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.277727 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93ba515d291b4c77ab4d37e26450d97d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.280962 5841 ts_tablet_manager.cc:1888] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.281085 5841 log.cc:1196] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:12:10.281428 5841 ts_tablet_manager.cc:1909] T 33a425a90f0449bcaaacb4c3f653d26c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.283074 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 778) took 5081 ms. Trace:
I20240507 08:12:10.283115 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 (DEFAULT_TABLE table=default.Table_cd9885f1f58049c599b7710b4e1f8977 [id=68bd0d8485c74121a3f338e2b6404584]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.283154 5841 rpcz_store.cc:276] 0507 08:12:05.201967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.203897 (+ 1930us) service_pool.cc:225] Handling call
0507 08:12:10.265796 (+5061899us) ts_tablet_manager.cc:1120] Deleting tablet 33a425a90f0449bcaaacb4c3f653d26c
0507 08:12:10.265805 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.270882 (+ 5077us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.280874 (+ 9992us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.283053 (+ 2179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5061702}
I20240507 08:12:10.283512 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 372dd1ea1a4043c9ad5afb1d48dfc133. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.284600 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 783) took 4962 ms. Trace:
I20240507 08:12:10.284673 5841 rpcz_store.cc:276] 0507 08:12:05.322588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.322894 (+ 306us) service_pool.cc:225] Handling call
0507 08:12:10.284455 (+4961561us) ts_tablet_manager.cc:1120] Deleting tablet 3769eb670ff442e2ad2fa5b6a934512a
0507 08:12:10.284467 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.284584 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4961359}
I20240507 08:12:10.285559 5841 tablet_replica.cc:330] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.285784 5841 raft_consensus.cc:2229] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.286088 5841 raft_consensus.cc:2258] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.286598 5841 ts_tablet_manager.cc:1875] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.288683 7044 raft_consensus.cc:482] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.288820 7044 raft_consensus.cc:504] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.289502 7044 leader_election.cc:288] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.289644 7044 raft_consensus.cc:482] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.289746 7044 raft_consensus.cc:504] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.290293 7044 leader_election.cc:288] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:10.291607 5520 leader_election.cc:341] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 168e49ae24314a7aae2291485ddeda44
W20240507 08:12:10.291628 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 96ef77ef4f474107bf31395f5f30c12f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96ef77ef4f474107bf31395f5f30c12f already in progress: creating tablet
W20240507 08:12:10.291877 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8a397c40278141b7bb80ea5fcdbb125b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8a397c40278141b7bb80ea5fcdbb125b already in progress: creating tablet
I20240507 08:12:10.291940 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 4452dbc20b47409f88d8859143f1ac17 (DEFAULT_TABLE table=default.Table_a79ee8acf6384968a6fffcbb9284c2eb [id=802677ef492642f28a0182514533f6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.292855 5686 tablet_service.cc:1467] Processing CreateTablet for tablet b2028f0db9104597adde7894ae14870c (DEFAULT_TABLE table=default.Table_2139500f65f9422c8a1662c61b6bb1c5 [id=521a98f62e84451a8a5cd95b0718aa41]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.293287 5689 tablet_service.cc:1467] Processing CreateTablet for tablet df515d25403645f29d186c611907b680 (DEFAULT_TABLE table=default.Table_c05cf821365c4b04a6992a5daf8cc22f [id=d8e4c8a0252745999746370ac26761b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.294186 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 38eb529e041148268d1deb653ce983c7 (DEFAULT_TABLE table=default.Table_d9cadcb91f984f24ac8f220b42514007 [id=9958b62f46684aa198cc0bbe6b880a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.294618 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 68981cf4d9e74d41b55c411b4c678dab (DEFAULT_TABLE table=default.Table_f1ba0da75f11411ab9babc78e508f6e8 [id=53eff278109c42259cc676e933380806]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.295754 7081 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:10.297047 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4452dbc20b47409f88d8859143f1ac17. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.302794 5841 ts_tablet_manager.cc:1888] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.304370 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b2028f0db9104597adde7894ae14870c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.305752 6934 raft_consensus.cc:482] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.306128 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df515d25403645f29d186c611907b680. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.306069 6934 raft_consensus.cc:504] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.306833 6934 leader_election.cc:288] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.307423 5841 log.cc:1196] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/97c978ec8bca4aa5b2ffaf22c3290747
I20240507 08:12:10.307698 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.308323 5841 ts_tablet_manager.cc:1909] T 97c978ec8bca4aa5b2ffaf22c3290747 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:10.308485 5378 leader_election.cc:341] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.309720 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 784) took 4967 ms. Trace:
I20240507 08:12:10.309798 5841 rpcz_store.cc:276] 0507 08:12:05.341884 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.343733 (+ 1849us) service_pool.cc:225] Handling call
0507 08:12:10.285517 (+4941784us) ts_tablet_manager.cc:1120] Deleting tablet 97c978ec8bca4aa5b2ffaf22c3290747
0507 08:12:10.285525 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.295626 (+ 10101us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.302710 (+ 7084us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.309703 (+ 6993us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4941599}
I20240507 08:12:10.311157 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 785) took 4969 ms. Trace:
I20240507 08:12:10.311228 5841 rpcz_store.cc:276] 0507 08:12:05.342028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.343961 (+ 1933us) service_pool.cc:225] Handling call
0507 08:12:10.311053 (+4967092us) ts_tablet_manager.cc:1120] Deleting tablet 055a30bfc61b4ff2be9f2418ca59b030
0507 08:12:10.311060 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.311144 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4966929}
I20240507 08:12:10.312052 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 786) took 4969 ms. Trace:
I20240507 08:12:10.312119 5841 rpcz_store.cc:276] 0507 08:12:05.342156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.344156 (+ 2000us) service_pool.cc:225] Handling call
0507 08:12:10.311960 (+4967804us) ts_tablet_manager.cc:1120] Deleting tablet 4ad95383487e458ca162038f6940271b
0507 08:12:10.311967 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.312039 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4967652}
I20240507 08:12:10.312785 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68981cf4d9e74d41b55c411b4c678dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.312949 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 787) took 4970 ms. Trace:
I20240507 08:12:10.313014 5841 rpcz_store.cc:276] 0507 08:12:05.342304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.344346 (+ 2042us) service_pool.cc:225] Handling call
0507 08:12:10.312810 (+4968464us) ts_tablet_manager.cc:1120] Deleting tablet b6ba3397e33b4cc7805a25e7535a383a
0507 08:12:10.312821 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.312931 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4968324}
I20240507 08:12:10.313858 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 788) took 4971 ms. Trace:
I20240507 08:12:10.313930 5841 rpcz_store.cc:276] 0507 08:12:05.342441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.344516 (+ 2075us) service_pool.cc:225] Handling call
0507 08:12:10.313776 (+4969260us) ts_tablet_manager.cc:1120] Deleting tablet 2bb648fe98e74248b5cd65603de58a02
0507 08:12:10.313783 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.313845 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4969089}
I20240507 08:12:10.314662 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 789) took 4899 ms. Trace:
I20240507 08:12:10.314726 5841 rpcz_store.cc:276] 0507 08:12:05.415338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.415396 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:10.314583 (+4899187us) ts_tablet_manager.cc:1120] Deleting tablet 93c68ddcfa7c40c180b6b12fabe165d6
0507 08:12:10.314590 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.314651 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4898992}
I20240507 08:12:10.315490 5841 tablet_replica.cc:330] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.315747 5841 raft_consensus.cc:2229] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.315966 5841 raft_consensus.cc:2258] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.316290 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eb529e041148268d1deb653ce983c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.317497 5841 ts_tablet_manager.cc:1875] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.321416 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.323539 7101 raft_consensus.cc:482] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.328066 5585 raft_consensus.cc:3037] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:10.328424 5377 leader_election.cc:341] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.328537 5377 leader_election.cc:302] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:10.329051 5520 leader_election.cc:341] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b4504b1ed3d449619926fe80469d9790
I20240507 08:12:10.329480 6934 raft_consensus.cc:2726] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.331557 7044 raft_consensus.cc:482] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.331713 7044 raft_consensus.cc:504] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.332401 7044 leader_election.cc:288] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:10.333911 5520 leader_election.cc:341] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7027b4cefa4d41659f819ee4326470c4
W20240507 08:12:10.326597 5654 consensus_peers.cc:485] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:10.322152 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 6549db5bfffe45d98930a2a615db6948 (DEFAULT_TABLE table=default.Table_f932b65c022243ba8acd49cc96e5dff1 [id=d051bd1329b041348df36aa4639f9442]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.339423 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6549db5bfffe45d98930a2a615db6948. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.340487 5583 raft_consensus.cc:3037] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.344167 7101 raft_consensus.cc:504] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.346089 5257 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:44738 (request call id 1164) took 1125 ms. Trace:
I20240507 08:12:10.336766 5840 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.346444 5840 ts_tablet_manager.cc:1367] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.123s user 0.000s sys 0.003s
I20240507 08:12:10.346210 5257 rpcz_store.cc:276] 0507 08:12:09.220135 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.125049 (+904914us) service_pool.cc:225] Handling call
0507 08:12:10.169659 (+ 44610us) spinlock_profiling.cc:243] Waited 44.3 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9c4cb3 00007f3bc39b62a6 00007f3bc39b5fd5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.170409 (+ 750us) catalog_manager.cc:2072] Acquired catalog manager lock
0507 08:12:10.170879 (+ 470us) catalog_manager.cc:2118] Created new table and tablet info
0507 08:12:10.345796 (+174917us) catalog_manager.cc:2178] Wrote table and tablets to system table
0507 08:12:10.345926 (+ 130us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0507 08:12:10.346030 (+ 104us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.214345 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.214460 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.214467 (+ 7us) write_op.cc:435] Acquired schema lock
0507 08:12:10.214471 (+ 4us) tablet.cc:660] Decoding operations
0507 08:12:10.214567 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.214577 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.214581 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:10.214646 (+ 65us) tablet.cc:699] Row locks acquired
0507 08:12:10.214648 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:10.214676 (+ 28us) write_op.cc:270] Start()
0507 08:12:10.214703 (+ 27us) write_op.cc:276] Timestamp: P: 1715069530214611 usec, L: 0
0507 08:12:10.214707 (+ 4us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.214849 (+ 142us) log.cc:842] Serialized 433 byte log entry
0507 08:12:10.344682 (+129833us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.344857 (+ 175us) write_op.cc:301] APPLY: starting
0507 08:12:10.344880 (+ 23us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.344934 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.344970 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.345102 (+ 132us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.345139 (+ 37us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0507 08:12:10.345147 (+ 8us) write_op.cc:312] APPLY: finished
0507 08:12:10.345242 (+ 95us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.345334 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.345358 (+ 24us) write_op.cc:454] Released schema lock
0507 08:12:10.345366 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.345437 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.345443 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":101918976,"child_traces":[["op",{"apply.queue_time_us":99,"apply.run_cpu_time_us":663,"apply.run_wall_time_us":664,"num_ops":2,"prepare.queue_time_us":43025,"prepare.run_cpu_time_us":597,"prepare.run_wall_time_us":977,"raft.queue_time_us":31,"raft.run_cpu_time_us":8,"raft.run_wall_time_us":9,"replication_time_us":129947}]]}
I20240507 08:12:10.347034 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a843afde2c19414c9b6c0581df4cb832" } new_table_name: "default.Table_548b730d6e06428c9f74b3cff2110d7d" modify_external_catalogs: true
I20240507 08:12:10.347436 5840 raft_consensus.cc:348] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.347656 5840 raft_consensus.cc:374] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.347733 5840 raft_consensus.cc:729] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.335788 6784 leader_election.cc:288] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.347937 5840 consensus_queue.cc:260] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.351670 7101 leader_election.cc:288] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.352628 5840 ts_tablet_manager.cc:1398] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:10.353099 5840 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.353471 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 777) took 4660 ms. Trace:
I20240507 08:12:10.353771 6513 rpcz_store.cc:276] 0507 08:12:05.692830 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.692883 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:10.181959 (+4489076us) ts_tablet_manager.cc:1120] Deleting tablet 077ccc3946b34008bea7b80d8aee2662
0507 08:12:10.185410 (+ 3451us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.250042 (+ 64632us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.254241 (+ 4199us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.353420 (+ 99179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4488935}
I20240507 08:12:10.354314 6513 tablet_replica.cc:330] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.354362 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]) successfully deleted
I20240507 08:12:10.354312 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.354604 6513 raft_consensus.cc:2229] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:10.354864 5654 leader_election.cc:341] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.355981 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:10.356410 5255 catalog_manager.cc:6554] ReplaceTablet: tablet aa094773f3bf418492bcd5b5021fd429 deleted and replaced by tablet 451f566fdf2a40a7a61a2d7ae38fffba
I20240507 08:12:10.356498 5255 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:44738 (request call id 1167) took 1025 ms. Trace:
I20240507 08:12:10.356909 5840 tablet_bootstrap.cc:654] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.356554 5255 rpcz_store.cc:276] 0507 08:12:09.331214 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.159963 (+828749us) service_pool.cc:225] Handling call
0507 08:12:10.167641 (+ 7678us) spinlock_profiling.cc:243] Waited 7.29 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9d105b 00007f3bc39c2be7 00007f3bc39c2915 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.356482 (+188841us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.219873 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.219949 (+ 76us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.219954 (+ 5us) write_op.cc:435] Acquired schema lock
0507 08:12:10.219956 (+ 2us) tablet.cc:660] Decoding operations
0507 08:12:10.220045 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.220056 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.220063 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:10.220129 (+ 66us) tablet.cc:699] Row locks acquired
0507 08:12:10.220131 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:10.220157 (+ 26us) write_op.cc:270] Start()
0507 08:12:10.220179 (+ 22us) write_op.cc:276] Timestamp: P: 1715069530220092 usec, L: 0
0507 08:12:10.220182 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.220295 (+ 113us) log.cc:842] Serialized 628 byte log entry
0507 08:12:10.352150 (+131855us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.353854 (+ 1704us) write_op.cc:301] APPLY: starting
0507 08:12:10.353864 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.353892 (+ 28us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.353916 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.353998 (+ 82us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.354045 (+ 47us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:12:10.354051 (+ 6us) write_op.cc:312] APPLY: finished
0507 08:12:10.354106 (+ 55us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.354144 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.354156 (+ 12us) write_op.cc:454] Released schema lock
0507 08:12:10.354161 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.354212 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.354215 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":16785536,"child_traces":[["op",{"apply.queue_time_us":1669,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":402,"num_ops":2,"prepare.queue_time_us":20789,"prepare.run_cpu_time_us":493,"prepare.run_wall_time_us":802,"raft.queue_time_us":244,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":131954}]]}
I20240507 08:12:10.357290 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_25e1a176f4084371aab7be282c6e6f6d" } new_table_name: "default.Table_4acf2380b98642d597ef313692c54ff6" modify_external_catalogs: true
I20240507 08:12:10.354871 6513 raft_consensus.cc:2258] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.358740 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d6f8f4205f1f48abaf4fcd7113ad44a8
I20240507 08:12:10.358744 6513 ts_tablet_manager.cc:1875] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.359243 5264 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:44738 (request call id 1161) took 1140 ms. Trace:
I20240507 08:12:10.359316 5264 rpcz_store.cc:276] 0507 08:12:09.218617 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.119236 (+900619us) service_pool.cc:225] Handling call
0507 08:12:10.173749 (+ 54513us) spinlock_profiling.cc:243] Waited 54.3 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9c5ca6 00007f3bc39b7f66 00007f3bc39b7c95 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.174152 (+ 403us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0507 08:12:10.174294 (+ 142us) catalog_manager.cc:2749] Modifying in-memory table state
0507 08:12:10.174367 (+ 73us) catalog_manager.cc:2758] Locking tablets
0507 08:12:10.200285 (+ 25918us) catalog_manager.cc:2772] Removing table and tablets from system table
0507 08:12:10.358555 (+158270us) catalog_manager.cc:2790] Removing table from by-name map
0507 08:12:10.358636 (+ 81us) catalog_manager.cc:2806] Committing in-memory state
0507 08:12:10.358684 (+ 48us) catalog_manager.cc:2810] Aborting table tasks
0507 08:12:10.359222 (+ 538us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.220696 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.220771 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.220775 (+ 4us) write_op.cc:435] Acquired schema lock
0507 08:12:10.220778 (+ 3us) tablet.cc:660] Decoding operations
0507 08:12:10.221757 (+ 979us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.221765 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.221770 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:10.221849 (+ 79us) tablet.cc:699] Row locks acquired
0507 08:12:10.221852 (+ 3us) write_op.cc:260] PREPARE: finished
0507 08:12:10.221876 (+ 24us) write_op.cc:270] Start()
0507 08:12:10.221898 (+ 22us) write_op.cc:276] Timestamp: P: 1715069530221811 usec, L: 0
0507 08:12:10.221901 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.222023 (+ 122us) log.cc:842] Serialized 762 byte log entry
0507 08:12:10.352190 (+130167us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.354282 (+ 2092us) write_op.cc:301] APPLY: starting
0507 08:12:10.354291 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.354321 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.354350 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.354412 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.354437 (+ 25us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0507 08:12:10.354443 (+ 6us) write_op.cc:312] APPLY: finished
0507 08:12:10.354489 (+ 46us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.354526 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.354552 (+ 26us) write_op.cc:454] Released schema lock
0507 08:12:10.354557 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.354604 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.354607 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":124844416,"child_traces":[["op",{"apply.queue_time_us":2067,"apply.run_cpu_time_us":365,"apply.run_wall_time_us":365,"num_ops":2,"prepare.queue_time_us":19724,"prepare.run_cpu_time_us":548,"prepare.run_wall_time_us":1662,"raft.queue_time_us":902,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":130275}]]}
I20240507 08:12:10.360141 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1b944c710a114e68a10b78be0d257898"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.360697 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b944c710a114e68a10b78be0d257898 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.361721 5260 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c] alter complete (version 1)
I20240507 08:12:10.363571 5840 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.363718 5840 ts_tablet_manager.cc:1367] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240507 08:12:10.364506 5840 raft_consensus.cc:348] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.364715 5840 raft_consensus.cc:374] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.364768 5840 raft_consensus.cc:729] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.364833 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5ee780b3d01a481b80949a3ade7e3cd3" } modify_external_catalogs: true
I20240507 08:12:10.365001 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5ee780b3d01a481b80949a3ade7e3cd3" } modify_external_catalogs: true
I20240507 08:12:10.364969 5840 consensus_queue.cc:260] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.365747 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7564834150854a7abd1fe3932c7e9e4b consensus state has no leader
I20240507 08:12:10.366071 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_452e7072fe42490f92423745f0bf9d07" } modify_external_catalogs: true
I20240507 08:12:10.366199 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_452e7072fe42490f92423745f0bf9d07" } modify_external_catalogs: true
I20240507 08:12:10.367964 5840 ts_tablet_manager.cc:1398] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:12:10.368219 5840 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.368330 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_760b66fcd51f489d8279e16c09e9955e" } modify_external_catalogs: true
I20240507 08:12:10.368460 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_760b66fcd51f489d8279e16c09e9955e" } modify_external_catalogs: true
I20240507 08:12:10.369768 5840 tablet_bootstrap.cc:654] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.371649 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.373765 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet aa094773f3bf418492bcd5b5021fd429 with delete_type TABLET_DATA_DELETED (replaced by tablet 451f566fdf2a40a7a61a2d7ae38fffba) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.376205 5262 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:44738 (request call id 1165) took 1155 ms. Trace:
I20240507 08:12:10.376312 5262 rpcz_store.cc:276] 0507 08:12:09.220224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.149686 (+929462us) service_pool.cc:225] Handling call
0507 08:12:10.179335 (+ 29649us) spinlock_profiling.cc:243] Waited 29.3 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9c4cb3 00007f3bc39b62a6 00007f3bc39b5fd5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.180051 (+ 716us) catalog_manager.cc:2072] Acquired catalog manager lock
0507 08:12:10.180491 (+ 440us) catalog_manager.cc:2118] Created new table and tablet info
0507 08:12:10.375936 (+195445us) catalog_manager.cc:2178] Wrote table and tablets to system table
0507 08:12:10.376075 (+ 139us) catalog_manager.cc:2212] Inserted table and tablets into CatalogManager maps
0507 08:12:10.376167 (+ 92us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.216314 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.216417 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.216422 (+ 5us) write_op.cc:435] Acquired schema lock
0507 08:12:10.216424 (+ 2us) tablet.cc:660] Decoding operations
0507 08:12:10.216487 (+ 63us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.216494 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.216499 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0507 08:12:10.216562 (+ 63us) tablet.cc:699] Row locks acquired
0507 08:12:10.216564 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:10.216585 (+ 21us) write_op.cc:270] Start()
0507 08:12:10.216614 (+ 29us) write_op.cc:276] Timestamp: P: 1715069530216522 usec, L: 0
0507 08:12:10.216617 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.216732 (+ 115us) log.cc:842] Serialized 433 byte log entry
0507 08:12:10.352054 (+135322us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.352923 (+ 869us) write_op.cc:301] APPLY: starting
0507 08:12:10.352934 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.352965 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.352994 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.353097 (+ 103us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.353128 (+ 31us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0507 08:12:10.353140 (+ 12us) write_op.cc:312] APPLY: finished
0507 08:12:10.353197 (+ 57us) log.cc:842] Serialized 29 byte log entry
0507 08:12:10.353242 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.353257 (+ 15us) write_op.cc:454] Released schema lock
0507 08:12:10.353262 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.353309 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.353312 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":67407104,"child_traces":[["op",{"apply.queue_time_us":838,"apply.run_cpu_time_us":426,"apply.run_wall_time_us":426,"num_ops":2,"prepare.queue_time_us":35415,"prepare.run_cpu_time_us":517,"prepare.run_wall_time_us":1666,"raft.queue_time_us":819,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":135422}]]}
I20240507 08:12:10.377187 6513 ts_tablet_manager.cc:1888] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 08:12:10.377281 6513 log.cc:1196] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c1c6b488b9014841899597d858bda5e2
I20240507 08:12:10.377142 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_50b030da7ecf4a98888919bc423e74a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.377712 6513 ts_tablet_manager.cc:1909] T c1c6b488b9014841899597d858bda5e2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:10.377745 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50b030da7ecf4a98888919bc423e74a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.378544 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 776) took 4685 ms. Trace:
I20240507 08:12:10.381629 6513 rpcz_store.cc:276] 0507 08:12:05.692723 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.693245 (+ 522us) service_pool.cc:225] Handling call
0507 08:12:10.354250 (+4661005us) ts_tablet_manager.cc:1120] Deleting tablet c1c6b488b9014841899597d858bda5e2
0507 08:12:10.354259 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.365218 (+ 10959us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.377060 (+ 11842us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.378523 (+ 1463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4660834}
I20240507 08:12:10.382280 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 784) took 4499 ms. Trace:
I20240507 08:12:10.382472 6513 rpcz_store.cc:276] 0507 08:12:05.882405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.887780 (+ 5375us) service_pool.cc:225] Handling call
0507 08:12:10.382125 (+4494345us) ts_tablet_manager.cc:1120] Deleting tablet 3618b416194f4effb29b823709593359
0507 08:12:10.382136 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.382254 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4494116}
I20240507 08:12:10.382841 6513 tablet_replica.cc:330] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.383085 6513 raft_consensus.cc:2229] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.383216 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9ccbe2e368d14b56859d62d1cc742f90
I20240507 08:12:10.383611 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 9ccbe2e368d14b56859d62d1cc742f90 deleted and replaced by tablet 096624b740fb4ac88a701a630c97b3eb
I20240507 08:12:10.383729 6513 raft_consensus.cc:2258] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.384382 6513 ts_tablet_manager.cc:1875] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.390964 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.391247 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 87377661d9374cef8ab1a97042b7f1b9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.391444 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ced879f35e14106b2609c78b7f24002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.391741 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet d45d3fbbe3764261a3a24523a77c6443 with delete_type TABLET_DATA_DELETED (replaced by tablet 2754fc46c36346f9828198e3bb6ed3a7) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.391911 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.392062 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 7027b4cefa4d41659f819ee4326470c4 with delete_type TABLET_DATA_DELETED (replaced by tablet 66247e08dfdf47f69ba24b074c8080f0) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.392228 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.392386 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:10.392565 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ff253ff827f4694bd27994825793108 consensus state has no leader
I20240507 08:12:10.392710 5263 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:44738 (request call id 1166) took 1168 ms. Trace:
I20240507 08:12:10.392766 5263 rpcz_store.cc:276] 0507 08:12:09.224719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.154262 (+929543us) service_pool.cc:225] Handling call
0507 08:12:10.177932 (+ 23670us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9c6f46 00007f3bc39b9c26 00007f3bc39b9955 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:10.178199 (+ 267us) catalog_manager.cc:2344] Looking up, locking, and authorizing table
0507 08:12:10.178431 (+ 232us) catalog_manager.cc:3613] Apply alter schema
0507 08:12:10.178588 (+ 157us) catalog_manager.cc:3636] Acquired catalog manager lock
0507 08:12:10.178665 (+ 77us) catalog_manager.cc:3783] Updating metadata on disk
0507 08:12:10.392369 (+213704us) catalog_manager.cc:3814] Committing alterations to in-memory state
0507 08:12:10.392694 (+ 325us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:10.215344 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:10.215448 (+ 104us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:10.215454 (+ 6us) write_op.cc:435] Acquired schema lock
0507 08:12:10.215458 (+ 4us) tablet.cc:660] Decoding operations
0507 08:12:10.215550 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:10.215559 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:10.215563 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:12:10.215610 (+ 47us) tablet.cc:699] Row locks acquired
0507 08:12:10.215613 (+ 3us) write_op.cc:260] PREPARE: finished
0507 08:12:10.215639 (+ 26us) write_op.cc:270] Start()
0507 08:12:10.215662 (+ 23us) write_op.cc:276] Timestamp: P: 1715069530215574 usec, L: 0
0507 08:12:10.215665 (+ 3us) op_driver.cc:331] REPLICATION: starting
0507 08:12:10.215783 (+ 118us) log.cc:842] Serialized 405 byte log entry
0507 08:12:10.351986 (+136203us) op_driver.cc:444] REPLICATION: finished
0507 08:12:10.352420 (+ 434us) write_op.cc:301] APPLY: starting
0507 08:12:10.352437 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:10.352480 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:10.352517 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:10.352567 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:10.352601 (+ 34us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:12:10.352608 (+ 7us) write_op.cc:312] APPLY: finished
0507 08:12:10.352687 (+ 79us) log.cc:842] Serialized 23 byte log entry
0507 08:12:10.352752 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.352770 (+ 18us) write_op.cc:454] Released schema lock
0507 08:12:10.352776 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:10.352831 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:10.352843 (+ 12us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":30023936,"child_traces":[["op",{"apply.queue_time_us":383,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":475,"num_ops":1,"prepare.queue_time_us":36233,"prepare.run_cpu_time_us":532,"prepare.run_wall_time_us":945,"raft.queue_time_us":33,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":136296}]]}
I20240507 08:12:10.393594 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e860ab852fe14a5993248381d5e7e17d" } modify_external_catalogs: true
I20240507 08:12:10.393749 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e860ab852fe14a5993248381d5e7e17d" } modify_external_catalogs: true
I20240507 08:12:10.397694 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet b4504b1ed3d449619926fe80469d9790 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.399098 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 with delete_type TABLET_DATA_DELETED (replaced by tablet 096624b740fb4ac88a701a630c97b3eb) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.399220 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 with delete_type TABLET_DATA_DELETED (replaced by tablet 096624b740fb4ac88a701a630c97b3eb) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.399323 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet dfa14901cdf14b72bea365b04ad521bd with delete_type TABLET_DATA_DELETED (replaced by tablet 3f01e3311a8b4553a31914922fb08d13) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.399528 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a425a90f0449bcaaacb4c3f653d26c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.399698 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4716c1285e91432a92b25ab13ada2779 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.399888 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet e555a89d340448bc852f2441ca850213 with delete_type TABLET_DATA_DELETED (replaced by tablet 62895377a8ac4506a2f4e10039dbdd74) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.400637 6513 ts_tablet_manager.cc:1888] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.400756 6513 log.cc:1196] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:10.400712 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_48b51787d9ce4ff1b9b8744ed86885bb [id=0a1bc06a351549d5aea075ad8d026476]) successfully deleted
W20240507 08:12:10.401008 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3618b416194f4effb29b823709593359 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3618b416194f4effb29b823709593359 already in progress: creating tablet
I20240507 08:12:10.401078 6513 ts_tablet_manager.cc:1909] T d45d3fbbe3764261a3a24523a77c6443 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.402256 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 787) took 4498 ms. Trace:
I20240507 08:12:10.402436 6513 rpcz_store.cc:276] 0507 08:12:05.903862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.903917 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:10.382795 (+4478878us) ts_tablet_manager.cc:1120] Deleting tablet d45d3fbbe3764261a3a24523a77c6443
0507 08:12:10.382806 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.389878 (+ 7072us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.400523 (+ 10645us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.402238 (+ 1715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4478712}
I20240507 08:12:10.403285 6513 tablet_replica.cc:330] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:10.404194 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 34adee00b529424b9f274c61ce73b0bc consensus state has no leader
I20240507 08:12:10.404603 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1971f4cb4ebc464586d955900fb585ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.405117 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1971f4cb4ebc464586d955900fb585ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.406399 6513 raft_consensus.cc:2229] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.406714 6513 pending_rounds.cc:62] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:10.406814 6513 pending_rounds.cc:68] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924781202993152 op_type: NO_OP noop_request { }
I20240507 08:12:10.406989 6513 raft_consensus.cc:2866] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:10.407124 6513 raft_consensus.cc:2258] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.407723 6513 ts_tablet_manager.cc:1875] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.409782 6934 raft_consensus.cc:482] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.409937 6934 raft_consensus.cc:504] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.410619 6934 leader_election.cc:288] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.411341 7079 tablet_replica.cc:330] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.411357 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.411540 5718 raft_consensus.cc:2454] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:10.411477 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.411559 7079 raft_consensus.cc:2229] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.411943 7079 raft_consensus.cc:2258] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.412014 5377 leader_election.cc:302] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:10.412339 6934 raft_consensus.cc:2781] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.412413 6934 raft_consensus.cc:482] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.412470 6934 raft_consensus.cc:3037] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.412500 7079 ts_tablet_manager.cc:1875] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:10.413556 5378 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.414279 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d45d3fbbe3764261a3a24523a77c6443 (table default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]) successfully deleted
W20240507 08:12:10.414587 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c245e220d07d4a3f9c98e0c38488f08a consensus state has no leader
I20240507 08:12:10.414980 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9766bca2ee764e50a876dff2ed050158"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.415445 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9766bca2ee764e50a876dff2ed050158 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.417305 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9e923448a06e4222b2ab2e13dcff9601" } new_table_name: "default.Table_5bc2fbd1f2a144369c08d69027aaf373" modify_external_catalogs: true
I20240507 08:12:10.419384 6513 ts_tablet_manager.cc:1888] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.419493 6513 log.cc:1196] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/070fdbf0b00b4582a184729c2f2a4b69
I20240507 08:12:10.420220 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6571d5b9dda642549111e4a278c03f33"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.420536 7103 log.cc:924] Time spent T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20240507 08:12:10.420881 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6571d5b9dda642549111e4a278c03f33 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.421165 6513 ts_tablet_manager.cc:1909] T 070fdbf0b00b4582a184729c2f2a4b69 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.422360 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 789) took 4518 ms. Trace:
I20240507 08:12:10.423920 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_709cc5845fc74e038578ef026cbb7567 [id=06457cc0f83a470a95273688e948d21a] alter complete (version 1)
I20240507 08:12:10.424062 6513 rpcz_store.cc:276] 0507 08:12:05.904067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.904137 (+ 70us) service_pool.cc:225] Handling call
0507 08:12:10.403238 (+4499101us) ts_tablet_manager.cc:1120] Deleting tablet 070fdbf0b00b4582a184729c2f2a4b69
0507 08:12:10.403247 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.412842 (+ 9595us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.419275 (+ 6433us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.422330 (+ 3055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4498968}
I20240507 08:12:10.424546 6513 tablet_replica.cc:330] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.424655 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ebd8a847575e430eb26365ca4cf1928a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.424782 6513 raft_consensus.cc:2229] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.425091 6513 raft_consensus.cc:2258] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:10.425176 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebd8a847575e430eb26365ca4cf1928a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.425797 6513 ts_tablet_manager.cc:1875] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.425987 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 070fdbf0b00b4582a184729c2f2a4b69 (table default.Table_b570163e3d2740c1abb9a44675630cfc [id=4851deb2bcf2430cb22de846f4410896]) successfully deleted
I20240507 08:12:10.426322 7079 ts_tablet_manager.cc:1888] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.426431 7079 log.cc:1196] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/070fdbf0b00b4582a184729c2f2a4b69
I20240507 08:12:10.426684 7079 ts_tablet_manager.cc:1909] T 070fdbf0b00b4582a184729c2f2a4b69 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.427212 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 070fdbf0b00b4582a184729c2f2a4b69 (table default.Table_b570163e3d2740c1abb9a44675630cfc [id=4851deb2bcf2430cb22de846f4410896]) successfully deleted
I20240507 08:12:10.428524 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.431253 7079 tablet_replica.cc:330] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.431461 7079 raft_consensus.cc:2229] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.431612 7079 raft_consensus.cc:2258] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.432081 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet a7df45c39d1b490db97c72a0eb6cf3ba with delete_type TABLET_DATA_DELETED (replaced by tablet 43225934cef04109b3dbc057b230f032) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.432343 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.432700 7079 ts_tablet_manager.cc:1875] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:10.432726 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.433804 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet aa094773f3bf418492bcd5b5021fd429 with delete_type TABLET_DATA_DELETED (replaced by tablet 451f566fdf2a40a7a61a2d7ae38fffba) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:10.437075 5654 consensus_peers.cc:485] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.440479 7079 ts_tablet_manager.cc:1888] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.440606 7079 log.cc:1196] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/87377661d9374cef8ab1a97042b7f1b9
I20240507 08:12:10.440982 7079 ts_tablet_manager.cc:1909] T 87377661d9374cef8ab1a97042b7f1b9 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.441707 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 87377661d9374cef8ab1a97042b7f1b9 (table default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]) successfully deleted
I20240507 08:12:10.442242 7079 tablet_replica.cc:330] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.442653 7079 raft_consensus.cc:2229] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.442844 7079 raft_consensus.cc:2258] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.443478 7079 ts_tablet_manager.cc:1875] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.444167 6513 ts_tablet_manager.cc:1888] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.444376 6513 log.cc:1196] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/193ec0b463d04835b74df80229c85cde
I20240507 08:12:10.444810 6513 ts_tablet_manager.cc:1909] T 193ec0b463d04835b74df80229c85cde P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.445700 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 790) took 4541 ms. Trace:
I20240507 08:12:10.445842 5839 consensus_queue.cc:260] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.448161 6513 rpcz_store.cc:276] 0507 08:12:05.904226 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.904289 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:10.424498 (+4520209us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:12:10.424508 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.436363 (+ 11855us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.444057 (+ 7694us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.445675 (+ 1618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4520079}
I20240507 08:12:10.448632 6513 tablet_replica.cc:330] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.448866 6513 raft_consensus.cc:2229] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.449446 6513 raft_consensus.cc:2258] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.450327 6513 ts_tablet_manager.cc:1875] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.451314 5841 ts_tablet_manager.cc:1888] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.451429 5841 log.cc:1196] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:12:10.451550 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 193ec0b463d04835b74df80229c85cde (table default.Table_f68ddcc012de4bd9ba219332fbfa8760 [id=3dd22e14be6b4addbb1bdbfd99a99ec7]) successfully deleted
I20240507 08:12:10.451792 5841 ts_tablet_manager.cc:1909] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.452628 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 790) took 5029 ms. Trace:
I20240507 08:12:10.452752 5841 rpcz_store.cc:276] 0507 08:12:05.423014 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.423063 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:10.315429 (+4892366us) ts_tablet_manager.cc:1120] Deleting tablet 7841f5ebf5ed4f6eb28dcbd4087f1713
0507 08:12:10.315442 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.350010 (+ 34568us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.451205 (+101195us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.452608 (+ 1403us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4892149}
I20240507 08:12:10.453123 5841 tablet_replica.cc:330] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.453414 5841 raft_consensus.cc:2229] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.451438 6934 raft_consensus.cc:504] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.454547 6934 leader_election.cc:288] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.454792 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:10.455418 5378 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.456095 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "565ac6091f5e48b1b024094d5560e275" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:10.456282 5718 raft_consensus.cc:3037] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.456663 7079 ts_tablet_manager.cc:1888] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.456756 7079 log.cc:1196] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/9ced879f35e14106b2609c78b7f24002
I20240507 08:12:10.457156 7079 ts_tablet_manager.cc:1909] T 9ced879f35e14106b2609c78b7f24002 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.457768 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 9ced879f35e14106b2609c78b7f24002 (table default.Table_dea0e2b1b090454cbd15a00f7e792d6e [id=aa09d786696945d5b4060819737a6e11]) successfully deleted
I20240507 08:12:10.458369 7079 tablet_replica.cc:330] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.458485 5841 raft_consensus.cc:2258] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.458554 7079 raft_consensus.cc:2229] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.458758 7079 pending_rounds.cc:62] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Trying to abort 1 pending ops.
I20240507 08:12:10.458801 7079 pending_rounds.cc:68] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924794672492544 op_type: NO_OP noop_request { }
I20240507 08:12:10.458896 7079 raft_consensus.cc:2866] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:10.458967 7079 raft_consensus.cc:2258] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.459048 5841 ts_tablet_manager.cc:1875] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.458947 5718 raft_consensus.cc:2454] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:10.460384 5377 leader_election.cc:302] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.460757 6934 raft_consensus.cc:2781] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.461833 7079 ts_tablet_manager.cc:1875] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.465631 6934 raft_consensus.cc:686] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:10.467183 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]). Expected version 1 got 0
I20240507 08:12:10.470362 5839 ts_tablet_manager.cc:1398] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.280s user 0.002s sys 0.000s
I20240507 08:12:10.472337 5839 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:10.473731 5839 tablet_bootstrap.cc:654] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.473842 6513 ts_tablet_manager.cc:1888] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.473938 6513 log.cc:1196] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/9e447562e82347a584e39586500c27c2
I20240507 08:12:10.474285 6513 ts_tablet_manager.cc:1909] T 9e447562e82347a584e39586500c27c2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.478346 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 788) took 4574 ms. Trace:
I20240507 08:12:10.479924 7079 ts_tablet_manager.cc:1888] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.483158 6513 rpcz_store.cc:276] 0507 08:12:05.903971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.904275 (+ 304us) service_pool.cc:225] Handling call
0507 08:12:10.448581 (+4544306us) ts_tablet_manager.cc:1120] Deleting tablet 9e447562e82347a584e39586500c27c2
0507 08:12:10.448593 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.470932 (+ 22339us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.473747 (+ 2815us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.478313 (+ 4566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4544153}
I20240507 08:12:10.483783 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 811) took 4470 ms. Trace:
I20240507 08:12:10.483865 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 9e447562e82347a584e39586500c27c2 (table default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]) successfully deleted
I20240507 08:12:10.483970 7079 log.cc:1196] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d45d3fbbe3764261a3a24523a77c6443
I20240507 08:12:10.484285 6513 rpcz_store.cc:276] 0507 08:12:06.013072 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.339998 (+326926us) service_pool.cc:225] Handling call
0507 08:12:10.483642 (+4143644us) ts_tablet_manager.cc:1120] Deleting tablet 12e85df8b900468ba2584fd8f2e9bb2d
0507 08:12:10.483653 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.483765 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4143436}
I20240507 08:12:10.484681 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 812) took 4374 ms. Trace:
I20240507 08:12:10.484762 6513 rpcz_store.cc:276] 0507 08:12:06.110242 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340222 (+229980us) service_pool.cc:225] Handling call
0507 08:12:10.484571 (+4144349us) ts_tablet_manager.cc:1120] Deleting tablet c7d19b7cc39a4337826819a20205aecc
0507 08:12:10.484581 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.484666 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4144178}
W20240507 08:12:10.484829 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 12e85df8b900468ba2584fd8f2e9bb2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e85df8b900468ba2584fd8f2e9bb2d already in progress: creating tablet
I20240507 08:12:10.485002 6513 tablet_replica.cc:330] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.485185 6513 raft_consensus.cc:2229] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:10.485445 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c7d19b7cc39a4337826819a20205aecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7d19b7cc39a4337826819a20205aecc already in progress: creating tablet
I20240507 08:12:10.485610 7079 ts_tablet_manager.cc:1909] T d45d3fbbe3764261a3a24523a77c6443 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.486040 7079 tablet_replica.cc:330] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.486214 7079 raft_consensus.cc:2229] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.486194 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d45d3fbbe3764261a3a24523a77c6443 (table default.Table_cb0dc1032c164b088e9d31013da8e43e [id=ab2b107148d34bed8067b24439bcf689]) successfully deleted
I20240507 08:12:10.486366 7079 raft_consensus.cc:2258] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.486883 7079 ts_tablet_manager.cc:1875] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.491762 6513 raft_consensus.cc:2258] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.492445 6513 ts_tablet_manager.cc:1875] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.495558 5255 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]). Expected version 1 got 0
I20240507 08:12:10.498302 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 2817163a934b4041bd139803ab6e0f8d (DEFAULT_TABLE table=default.Table_0f7492244d9a44b3998beb1d5a959eeb [id=383c9ba314894e2d97650da7327c6b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.498450 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 411619d6fb934f63b059ae7a489c8bf8 (DEFAULT_TABLE table=default.Table_a34af5c61e5649e5abc9c88235891674 [id=8b27bd9862864cafa09a94326eaa3218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.498721 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2817163a934b4041bd139803ab6e0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.499670 5840 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.499794 5840 ts_tablet_manager.cc:1367] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.132s user 0.000s sys 0.006s
I20240507 08:12:10.500403 5840 raft_consensus.cc:348] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.500562 5840 raft_consensus.cc:374] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.500602 5840 raft_consensus.cc:729] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.500775 5840 consensus_queue.cc:260] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.502046 5654 consensus_peers.cc:485] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.502338 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411619d6fb934f63b059ae7a489c8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.504832 7079 ts_tablet_manager.cc:1888] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.504926 7079 log.cc:1196] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/168e49ae24314a7aae2291485ddeda44
I20240507 08:12:10.505184 7079 ts_tablet_manager.cc:1909] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.505885 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 168e49ae24314a7aae2291485ddeda44 (table default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]) successfully deleted
W20240507 08:12:10.506441 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5934f03ca4a949b79998b8606b92f440 consensus state has no leader
I20240507 08:12:10.508270 6513 ts_tablet_manager.cc:1888] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.510759 6513 log.cc:1196] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/9ced879f35e14106b2609c78b7f24002
I20240507 08:12:10.511379 6513 ts_tablet_manager.cc:1909] T 9ced879f35e14106b2609c78b7f24002 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.511595 7079 tablet_replica.cc:330] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:10.509877 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 271182us
I20240507 08:12:10.512629 7079 raft_consensus.cc:2229] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.512866 7079 raft_consensus.cc:2258] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.513235 6508 consensus_queue.cc:1035] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.513839 7079 ts_tablet_manager.cc:1875] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.514286 5656 leader_election.cc:302] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.514591 6508 raft_consensus.cc:2781] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.514698 6508 raft_consensus.cc:686] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.514920 6508 consensus_queue.cc:237] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.515355 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 2817163a934b4041bd139803ab6e0f8d (DEFAULT_TABLE table=default.Table_0f7492244d9a44b3998beb1d5a959eeb [id=383c9ba314894e2d97650da7327c6b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.515714 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2817163a934b4041bd139803ab6e0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.517664 5562 tablet_service.cc:1467] Processing CreateTablet for tablet e241dd3c1af14b278b1632481edd0f30 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:10.518072 5517 leader_election.cc:341] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:12:10.518190 5517 leader_election.cc:302] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.518757 5517 leader_election.cc:302] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.519244 6886 raft_consensus.cc:2726] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.519393 6886 raft_consensus.cc:2781] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.519444 6886 raft_consensus.cc:482] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.519510 6886 raft_consensus.cc:3037] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.520804 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 5934f03ca4a949b79998b8606b92f440 (DEFAULT_TABLE table=default.Table_5bc2fbd1f2a144369c08d69027aaf373 [id=794b5a14afe14608b5e50295fbc585f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.521351 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 411619d6fb934f63b059ae7a489c8bf8 (DEFAULT_TABLE table=default.Table_a34af5c61e5649e5abc9c88235891674 [id=8b27bd9862864cafa09a94326eaa3218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.521427 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 50358f99145c476fb3b0f8f5e2487aab (DEFAULT_TABLE table=default.Table_6991af2fca624791af1a67498a53b156 [id=910107a4e3854212a3cad0ddb4888b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.521517 5550 tablet_service.cc:1467] Processing CreateTablet for tablet a8adad098c934d11982d39aa1cf8b4bc (DEFAULT_TABLE table=default.Table_c39926d459d14c72884a82bf549db3ff [id=7c8f64d9bf2b48b5abba9f9ed53b31e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.522581 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411619d6fb934f63b059ae7a489c8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.524430 5839 tablet_bootstrap.cc:492] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:10.524585 5839 ts_tablet_manager.cc:1367] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.003s
I20240507 08:12:10.525698 5839 raft_consensus.cc:348] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.525840 5696 tablet_service.cc:1467] Processing CreateTablet for tablet a8adad098c934d11982d39aa1cf8b4bc (DEFAULT_TABLE table=default.Table_c39926d459d14c72884a82bf549db3ff [id=7c8f64d9bf2b48b5abba9f9ed53b31e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.525991 5839 raft_consensus.cc:374] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.526067 5839 raft_consensus.cc:729] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.526175 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8adad098c934d11982d39aa1cf8b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.526499 5583 raft_consensus.cc:3037] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.526338 5839 consensus_queue.cc:260] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.527014 5585 raft_consensus.cc:3037] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.527302 5839 ts_tablet_manager.cc:1398] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:10.528581 6934 consensus_queue.cc:237] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.529120 5654 consensus_peers.cc:485] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.529310 7101 raft_consensus.cc:482] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.529417 7101 raft_consensus.cc:504] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.529852 5686 tablet_service.cc:1467] Processing CreateTablet for tablet b7943787b81d45f08cada28f13694f66 (DEFAULT_TABLE table=default.Table_0f33de0d21b1417680053925d89ecf91 [id=2724329ee1f1487bb0ff3b9aaab9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.530014 7101 leader_election.cc:288] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.530177 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7943787b81d45f08cada28f13694f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.533409 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.533887 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 5934f03ca4a949b79998b8606b92f440 (DEFAULT_TABLE table=default.Table_5bc2fbd1f2a144369c08d69027aaf373 [id=794b5a14afe14608b5e50295fbc585f0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:10.534055 5654 leader_election.cc:341] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.534210 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5934f03ca4a949b79998b8606b92f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.535357 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.535543 5441 raft_consensus.cc:2454] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.535842 5441 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.535981 5441 raft_consensus.cc:2454] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.536280 5840 ts_tablet_manager.cc:1398] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.036s user 0.000s sys 0.002s
I20240507 08:12:10.536603 5840 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.536739 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d617213b959452ab6519809e56db128" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.536895 5444 raft_consensus.cc:3037] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.537652 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4504b1ed3d449619926fe80469d9790" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.537997 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7027b4cefa4d41659f819ee4326470c4" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.538141 5719 raft_consensus.cc:2454] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
I20240507 08:12:10.538270 5840 tablet_bootstrap.cc:654] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.538404 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.538542 5445 raft_consensus.cc:3037] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.539501 5444 raft_consensus.cc:2454] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.540897 5445 raft_consensus.cc:2454] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.541553 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:10.541604 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "168e49ae24314a7aae2291485ddeda44" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.542294 5695 tablet_service.cc:1467] Processing CreateTablet for tablet e241dd3c1af14b278b1632481edd0f30 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.542640 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e241dd3c1af14b278b1632481edd0f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.542519 7055 tablet.cc:1722] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240507 08:12:10.545167 5654 consensus_peers.cc:485] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.551318 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b98946e635c74c888d87a7ed5a3437c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.551939 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b98946e635c74c888d87a7ed5a3437c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.558957 6886 raft_consensus.cc:504] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.559675 6886 leader_election.cc:288] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.559266 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 50358f99145c476fb3b0f8f5e2487aab (DEFAULT_TABLE table=default.Table_6991af2fca624791af1a67498a53b156 [id=910107a4e3854212a3cad0ddb4888b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.560534 5565 tablet_service.cc:1467] Processing CreateTablet for tablet b7943787b81d45f08cada28f13694f66 (DEFAULT_TABLE table=default.Table_0f33de0d21b1417680053925d89ecf91 [id=2724329ee1f1487bb0ff3b9aaab9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.560845 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50358f99145c476fb3b0f8f5e2487aab. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:10.561407 5520 leader_election.cc:341] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e555a89d340448bc852f2441ca850213
I20240507 08:12:10.512351 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 813) took 4378 ms. Trace:
I20240507 08:12:10.564019 6513 rpcz_store.cc:276] 0507 08:12:06.134031 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340408 (+206377us) service_pool.cc:225] Handling call
0507 08:12:10.484963 (+4144555us) ts_tablet_manager.cc:1120] Deleting tablet 9ced879f35e14106b2609c78b7f24002
0507 08:12:10.484970 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.501046 (+ 16076us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.508167 (+ 7121us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.512332 (+ 4165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4144426}
I20240507 08:12:10.564918 5517 leader_election.cc:302] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.565326 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 814) took 4280 ms. Trace:
I20240507 08:12:10.566896 5517 leader_election.cc:302] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:10.567099 5517 raft_consensus.cc:2670] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: unable to run election callback: Service unavailable: Thread pool token was shut down
I20240507 08:12:10.567497 6934 raft_consensus.cc:482] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.567921 6934 raft_consensus.cc:504] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.567087 6513 rpcz_store.cc:276] 0507 08:12:06.285359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340548 (+ 55189us) service_pool.cc:225] Handling call
0507 08:12:10.565205 (+4224657us) ts_tablet_manager.cc:1120] Deleting tablet 6def4bf6a7554a4090a39b1823c4097c
0507 08:12:10.565215 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.565308 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4224535}
I20240507 08:12:10.569898 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 815) took 4284 ms. Trace:
I20240507 08:12:10.569083 6934 leader_election.cc:288] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.570683 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.571141 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:10.571411 5517 leader_election.cc:302] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:10.571578 5517 raft_consensus.cc:2670] T 168e49ae24314a7aae2291485ddeda44 P fe673739d8084e7da5d7edd722e857d8: unable to run election callback: Service unavailable: Thread pool token was shut down
W20240507 08:12:10.572822 5378 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.572773 6513 rpcz_store.cc:276] 0507 08:12:06.285452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340682 (+ 55230us) service_pool.cc:225] Handling call
0507 08:12:10.569779 (+4229097us) ts_tablet_manager.cc:1120] Deleting tablet d33a3ccd7a5e4b9e956735443d4c484a
0507 08:12:10.569788 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.569880 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4228961}
I20240507 08:12:10.573071 6513 tablet_replica.cc:330] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.573081 5841 ts_tablet_manager.cc:1888] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.574086 6513 raft_consensus.cc:2229] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.574213 5841 log.cc:1196] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d451d03349794726a71c255a7f177436
I20240507 08:12:10.573354 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.570443 5840 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.575330 5840 ts_tablet_manager.cc:1367] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
W20240507 08:12:10.575323 5377 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.575438 5377 leader_election.cc:302] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.575907 6934 raft_consensus.cc:2726] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.576135 5840 raft_consensus.cc:348] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.576377 5840 raft_consensus.cc:374] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.576478 5840 raft_consensus.cc:729] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.577710 7101 raft_consensus.cc:482] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.578411 7101 raft_consensus.cc:504] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.579026 7101 leader_election.cc:288] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.577664 6513 raft_consensus.cc:2258] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.579619 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2df872d04b94c30872b03367df3cc31" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:10.579856 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:10.580070 6513 ts_tablet_manager.cc:1875] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:10.580210 5654 leader_election.cc:341] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.581413 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 9ced879f35e14106b2609c78b7f24002 (table default.Table_dea0e2b1b090454cbd15a00f7e792d6e [id=aa09d786696945d5b4060819737a6e11]) successfully deleted
W20240507 08:12:10.582401 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6def4bf6a7554a4090a39b1823c4097c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6def4bf6a7554a4090a39b1823c4097c already in progress: creating tablet
W20240507 08:12:10.583647 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d33a3ccd7a5e4b9e956735443d4c484a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d33a3ccd7a5e4b9e956735443d4c484a already in progress: creating tablet
I20240507 08:12:10.576716 5840 consensus_queue.cc:260] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.586999 5840 ts_tablet_manager.cc:1398] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240507 08:12:10.581349 5264 catalog_manager.cc:5484] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:10.588974 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:10.590431 5841 ts_tablet_manager.cc:1909] T d451d03349794726a71c255a7f177436 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.592908 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e555a89d340448bc852f2441ca850213" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:10.594269 5719 raft_consensus.cc:3037] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.597039 5719 raft_consensus.cc:2454] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:12:10.598357 7101 consensus_queue.cc:1035] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.518075 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e241dd3c1af14b278b1632481edd0f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.599512 5840 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.601083 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5934f03ca4a949b79998b8606b92f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.602217 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50358f99145c476fb3b0f8f5e2487aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.604342 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8adad098c934d11982d39aa1cf8b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.609709 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7943787b81d45f08cada28f13694f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.600349 5843 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:10.610093 5843 ts_tablet_manager.cc:1367] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.470s user 0.006s sys 0.001s
I20240507 08:12:10.610813 5843 raft_consensus.cc:348] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.610993 5843 raft_consensus.cc:374] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.611040 5843 raft_consensus.cc:729] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.599453 5656 leader_election.cc:302] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.612185 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 791) took 5143 ms. Trace:
I20240507 08:12:10.612262 5841 rpcz_store.cc:276] 0507 08:12:05.468630 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.468700 (+ 70us) service_pool.cc:225] Handling call
0507 08:12:10.453079 (+4984379us) ts_tablet_manager.cc:1120] Deleting tablet d451d03349794726a71c255a7f177436
0507 08:12:10.453088 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.533766 (+ 80678us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.571807 (+ 38041us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.612162 (+ 40355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4984161}
I20240507 08:12:10.618078 5840 tablet_bootstrap.cc:654] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.618480 7095 raft_consensus.cc:2781] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.618559 7095 raft_consensus.cc:482] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.618633 7095 raft_consensus.cc:3037] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.619717 7079 ts_tablet_manager.cc:1888] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.619809 7079 log.cc:1196] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/7027b4cefa4d41659f819ee4326470c4
I20240507 08:12:10.620070 7079 ts_tablet_manager.cc:1909] T 7027b4cefa4d41659f819ee4326470c4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.620625 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 7027b4cefa4d41659f819ee4326470c4 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]) successfully deleted
I20240507 08:12:10.621109 7095 raft_consensus.cc:504] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.622129 7095 leader_election.cc:288] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.623749 7079 tablet_replica.cc:330] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.624480 7079 raft_consensus.cc:2229] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.624730 7079 raft_consensus.cc:2258] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.626865 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_14fe89e6f91f4d1593dde48abd8bebda" } modify_external_catalogs: true
I20240507 08:12:10.627130 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_14fe89e6f91f4d1593dde48abd8bebda" } modify_external_catalogs: true
I20240507 08:12:10.628499 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f0ed0e76d4c746c189b43ebcf1c271b5" } new_table_name: "default.Table_599be0c8696c4c4f94c232b1668f717c" modify_external_catalogs: true
I20240507 08:12:10.629807 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:10.630280 6513 ts_tablet_manager.cc:1888] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.630753 6513 log.cc:1196] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/7027b4cefa4d41659f819ee4326470c4
W20240507 08:12:10.630414 5654 leader_election.cc:341] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.630312 5656 leader_election.cc:302] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.631244 7095 raft_consensus.cc:2781] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.631346 7095 raft_consensus.cc:482] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.631453 7095 raft_consensus.cc:3037] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.634256 7095 raft_consensus.cc:504] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.634598 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 34adee00b529424b9f274c61ce73b0bc consensus state has no leader
I20240507 08:12:10.635012 7095 leader_election.cc:288] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.635213 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 792) took 5086 ms. Trace:
I20240507 08:12:10.635659 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:10.636127 5654 leader_election.cc:341] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.636109 5841 rpcz_store.cc:276] 0507 08:12:05.547516 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.547582 (+ 66us) service_pool.cc:225] Handling call
0507 08:12:10.633908 (+5086326us) ts_tablet_manager.cc:1120] Deleting tablet 2b13fbddcab5460db69f43e63f490fba
0507 08:12:10.633921 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.634042 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5086061}
W20240507 08:12:10.636989 5656 consensus_peers.cc:485] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7841f5ebf5ed4f6eb28dcbd4087f1713. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.637213 5656 leader_election.cc:302] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.637846 7095 raft_consensus.cc:2781] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.637990 7095 raft_consensus.cc:686] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.638231 7095 consensus_queue.cc:237] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.642585 6508 consensus_queue.cc:1035] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.643368 6513 ts_tablet_manager.cc:1909] T 7027b4cefa4d41659f819ee4326470c4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.643738 6784 consensus_queue.cc:1035] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.643899 6508 consensus_queue.cc:1035] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.644104 5656 leader_election.cc:302] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.645848 6508 raft_consensus.cc:2781] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.645848 7044 raft_consensus.cc:2726] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.645972 6508 raft_consensus.cc:686] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.646196 6508 consensus_queue.cc:237] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.648068 7079 ts_tablet_manager.cc:1875] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.653762 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_dffc373d1b8940d6bdcc3280f92d0743 [id=c59b446f65c842478dae29b4065fce5a] alter complete (version 2)
I20240507 08:12:10.655215 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 81fcf82e60c94e9eaadd75bd37595261
I20240507 08:12:10.656070 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.656441 7044 raft_consensus.cc:482] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.656556 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.656591 7044 raft_consensus.cc:504] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.657265 7044 leader_election.cc:288] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:10.657763 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 39e5e67df91f4029bf55e5a82dc31f82
I20240507 08:12:10.658963 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.659003 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 816) took 4373 ms. Trace:
I20240507 08:12:10.659122 6513 rpcz_store.cc:276] 0507 08:12:06.285528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340757 (+ 55229us) service_pool.cc:225] Handling call
0507 08:12:10.573026 (+4232269us) ts_tablet_manager.cc:1120] Deleting tablet 7027b4cefa4d41659f819ee4326470c4
0507 08:12:10.573035 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.614111 (+ 41076us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.630174 (+ 16063us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.658984 (+ 28810us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12467,"tablet-delete.queue_time_us":4232041}
I20240507 08:12:10.659462 6513 tablet_replica.cc:330] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.660954 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 924d8d2fe2f148cba13f107e4c70fc68
I20240507 08:12:10.660933 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 7027b4cefa4d41659f819ee4326470c4 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]) successfully deleted
I20240507 08:12:10.661473 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 128c2034d917473eb5c496319fdc9625
I20240507 08:12:10.661065 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.661623 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.662395 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c2034d917473eb5c496319fdc9625 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.662436 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c2034d917473eb5c496319fdc9625 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.662616 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:10.662817 5656 consensus_peers.cc:485] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302fbf120e9d472ba6bdf9e6cbdf6dbd. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.663530 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4c741b9eb00a4e27895027010bdb5ae7" } new_table_name: "default.Table_11b76a3861af4745a4b2fd1b2bcd6478" modify_external_catalogs: true
I20240507 08:12:10.663013 6513 raft_consensus.cc:2229] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.663983 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9d3bcb49239048a198510a548f153eaf" } new_table_name: "default.Table_5cc40acfd8e04ef1813f07ec723496d6" modify_external_catalogs: true
I20240507 08:12:10.665463 6513 raft_consensus.cc:2258] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.665452 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e9226c81998d47d7a3359930560d2953"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.666067 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9226c81998d47d7a3359930560d2953 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.666093 6513 ts_tablet_manager.cc:1875] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.667307 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_089a4f3cc7ad43499020b5c9d1e6f969"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.667907 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_089a4f3cc7ad43499020b5c9d1e6f969 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.673749 5517 leader_election.cc:302] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.674365 7044 raft_consensus.cc:2781] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.674500 7044 raft_consensus.cc:686] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:12:10.674734 7044 consensus_queue.cc:237] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.679659 5654 consensus_peers.cc:485] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.682457 5583 raft_consensus.cc:3037] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:10.685058 5520 leader_election.cc:341] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: a7df45c39d1b490db97c72a0eb6cf3ba
I20240507 08:12:10.685832 7055 tablet.cc:1722] T 5782cfa5eca74c19a2e724a9124597f2 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20240507 08:12:10.697824 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6a25b41cf0d45f9bc3b2d9731a10da3 consensus state has no leader
W20240507 08:12:10.699177 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38fc0180e9d34d11885b819529407930 consensus state has no leader
I20240507 08:12:10.699254 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f0de4c39e4294c98aeb782a829a89f68" } new_table_name: "default.Table_964680dde8ab415a8553e6ac5bdb6a72" modify_external_catalogs: true
I20240507 08:12:10.699627 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bb71acdadad640e7a4b25604ae4f80fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.700268 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb71acdadad640e7a4b25604ae4f80fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.700584 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ff3b0a8492a246b3be16662405bc7a09"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.701546 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff3b0a8492a246b3be16662405bc7a09 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.702836 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_92a6805df9e7430dbaff524aaafe783a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.703342 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_92a6805df9e7430dbaff524aaafe783a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.706830 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.706964 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707042 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707191 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707202 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707371 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707491 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.707644 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ced879f35e14106b2609c78b7f24002 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.710561 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e447562e82347a584e39586500c27c2 with delete_type TABLET_DATA_DELETED (replaced by tablet 7d89e4185229431f9a68f599da4aff34) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.710939 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.711305 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.711644 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 077ccc3946b34008bea7b80d8aee2662 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.711992 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 193ec0b463d04835b74df80229c85cde with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.712337 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 79524df2415146ab926425c42068626c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.712678 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.713100 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 070fdbf0b00b4582a184729c2f2a4b69 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:10.672997 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2df872d04b94c30872b03367df3cc31" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:10.713752 5446 raft_consensus.cc:2454] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:10.692900 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.714004 5437 raft_consensus.cc:3037] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.714252 5841 tablet_replica.cc:330] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.714421 5841 raft_consensus.cc:2229] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.714565 5841 raft_consensus.cc:2258] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.715075 5841 ts_tablet_manager.cc:1875] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.716572 5437 raft_consensus.cc:2454] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.719859 5839 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:10.721153 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.707677 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6e68364238c4b66b8c0de0f4a2ea855" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.721843 5447 raft_consensus.cc:3037] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.721972 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5c49745fce2949d1909baf7583b157a7
I20240507 08:12:10.713322 5843 consensus_queue.cc:260] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.724030 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c49745fce2949d1909baf7583b157a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:10.724371 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c053869c840e4773a3c8e07a8b426b28 consensus state has no leader
I20240507 08:12:10.724418 5447 raft_consensus.cc:2454] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.724953 5843 ts_tablet_manager.cc:1398] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.115s user 0.001s sys 0.001s
I20240507 08:12:10.727527 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3c747008c670486b8d63c755d6ca129b" } new_table_name: "default.Table_58f0a0f4d9b74721a90f067d9bb7ef12" modify_external_catalogs: true
I20240507 08:12:10.731961 5843 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:10.734138 5378 consensus_peers.cc:485] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:10.734416 5654 consensus_peers.cc:485] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.736369 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb6e602d297c493fa3c7969a29ea399b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.740124 5656 leader_election.cc:302] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.740793 6784 raft_consensus.cc:2781] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:10.740876 6784 raft_consensus.cc:482] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:10.740936 6784 raft_consensus.cc:3037] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.741240 5840 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.741346 5840 ts_tablet_manager.cc:1367] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.142s user 0.006s sys 0.004s
W20240507 08:12:10.741469 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb6e602d297c493fa3c7969a29ea399b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.743683 6784 raft_consensus.cc:504] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.744423 6784 leader_election.cc:288] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:10.745517 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 consensus state has no leader
I20240507 08:12:10.748911 5843 tablet_bootstrap.cc:654] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:10.750747 5251 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 458850us
W20240507 08:12:10.751107 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4e3ea4658bb946439e4a50962444b3e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e3ea4658bb946439e4a50962444b3e2 already in progress: creating tablet
I20240507 08:12:10.751391 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 33a425a90f0449bcaaacb4c3f653d26c (table default.Table_81657f34e12b4ca0b2fd8ee26ab0e663 [id=b1f2776824d24099bbc0110c3da97c2e]) successfully deleted
I20240507 08:12:10.722643 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c49745fce2949d1909baf7583b157a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:10.739037 5517 leader_election.cc:341] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: a7df45c39d1b490db97c72a0eb6cf3ba
I20240507 08:12:10.752333 5517 leader_election.cc:302] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:10.752561 6886 raft_consensus.cc:2726] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.721407 5839 tablet_bootstrap.cc:654] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.753263 5656 leader_election.cc:302] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.745321 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2df872d04b94c30872b03367df3cc31" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:10.753729 6784 raft_consensus.cc:2781] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.753852 6784 raft_consensus.cc:686] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.754120 6784 consensus_queue.cc:237] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:10.756834 5654 leader_election.cc:341] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.764950 7079 ts_tablet_manager.cc:1888] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.765146 7079 log.cc:1196] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:12:10.765733 7079 ts_tablet_manager.cc:1909] T b7947ca2d8324dbb9c909ec50852fed8 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.769166 5840 raft_consensus.cc:348] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.769397 5840 raft_consensus.cc:374] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.769459 5840 raft_consensus.cc:729] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.769531 5656 leader_election.cc:302] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:10.769690 5840 consensus_queue.cc:260] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.770170 5840 ts_tablet_manager.cc:1398] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240507 08:12:10.771034 5840 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.772758 5840 tablet_bootstrap.cc:654] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.772840 7079 tablet_replica.cc:330] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.773082 7079 raft_consensus.cc:2229] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.774771 7079 raft_consensus.cc:2258] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.775696 7079 ts_tablet_manager.cc:1875] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.782033 7079 ts_tablet_manager.cc:1888] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.782303 7079 log.cc:1196] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b4504b1ed3d449619926fe80469d9790
I20240507 08:12:10.782727 7079 ts_tablet_manager.cc:1909] T b4504b1ed3d449619926fe80469d9790 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.783574 7079 tablet_replica.cc:330] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.783794 7079 raft_consensus.cc:2229] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.783982 7079 raft_consensus.cc:2258] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.784554 7079 ts_tablet_manager.cc:1875] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.804628 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b2df872d04b94c30872b03367df3cc31" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:10.804886 5439 raft_consensus.cc:3037] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:10.811758 5439 raft_consensus.cc:2454] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:10.815542 7079 ts_tablet_manager.cc:1888] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.815696 7079 log.cc:1196] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/dfa14901cdf14b72bea365b04ad521bd
I20240507 08:12:10.818820 7079 ts_tablet_manager.cc:1909] T dfa14901cdf14b72bea365b04ad521bd P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.825785 7079 tablet_replica.cc:330] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.826059 7079 raft_consensus.cc:2229] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.826270 7079 raft_consensus.cc:2258] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.826810 7079 ts_tablet_manager.cc:1875] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.829758 5839 tablet_bootstrap.cc:492] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:10.830008 5839 ts_tablet_manager.cc:1367] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.110s user 0.002s sys 0.001s
I20240507 08:12:10.830884 5839 raft_consensus.cc:348] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.831146 5839 raft_consensus.cc:374] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.831243 5839 raft_consensus.cc:729] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.831508 5839 consensus_queue.cc:260] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.832052 5839 ts_tablet_manager.cc:1398] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:10.833369 7079 ts_tablet_manager.cc:1888] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.833465 7079 log.cc:1196] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:12:10.835544 7107 consensus_queue.cc:1035] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.835714 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a51f8aaf8b364d69b4f79ed4cceb5157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.841821 7079 ts_tablet_manager.cc:1909] T 33a425a90f0449bcaaacb4c3f653d26c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.821805 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b7947ca2d8324dbb9c909ec50852fed8 (table default.Table_f7d324ad4a784569bdf917a46203715f [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:12:10.852267 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet c43f3de489904f2ba2a637d408619bfc from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:10.854033 5843 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:10.854769 5843 ts_tablet_manager.cc:1367] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.123s user 0.001s sys 0.005s
I20240507 08:12:10.856710 5843 raft_consensus.cc:348] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.776074 6675 raft_consensus.cc:2781] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.857100 6675 raft_consensus.cc:686] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.857394 6675 consensus_queue.cc:237] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.790624 6513 ts_tablet_manager.cc:1888] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.858455 6513 log.cc:1196] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b7947ca2d8324dbb9c909ec50852fed8
I20240507 08:12:10.858661 5839 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:10.858853 6513 ts_tablet_manager.cc:1909] T b7947ca2d8324dbb9c909ec50852fed8 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.859602 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 819) took 4557 ms. Trace:
I20240507 08:12:10.859684 6513 rpcz_store.cc:276] 0507 08:12:06.301716 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.344734 (+ 43018us) service_pool.cc:225] Handling call
0507 08:12:10.659409 (+4314675us) ts_tablet_manager.cc:1120] Deleting tablet b7947ca2d8324dbb9c909ec50852fed8
0507 08:12:10.659422 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.669065 (+ 9643us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.790517 (+121452us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.859580 (+ 69063us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1172,"tablet-delete.queue_time_us":4314464}
I20240507 08:12:10.859990 6513 tablet_replica.cc:330] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.860158 6513 raft_consensus.cc:2229] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.860448 6513 raft_consensus.cc:2258] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.860595 7079 tablet_replica.cc:330] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.860756 7079 raft_consensus.cc:2229] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.860884 7079 raft_consensus.cc:2258] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.860965 6513 ts_tablet_manager.cc:1875] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.861333 7079 ts_tablet_manager.cc:1875] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.817445 5656 leader_election.cc:302] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:10.861829 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 44423.9us
I20240507 08:12:10.861793 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5e958b4da1dd43ec9cc03a7554860b1c" } modify_external_catalogs: true
W20240507 08:12:10.862164 5656 consensus_peers.cc:485] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.817714 6508 consensus_queue.cc:1035] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.862771 6508 raft_consensus.cc:2781] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:10.862872 6508 raft_consensus.cc:686] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:10.863062 6508 consensus_queue.cc:237] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.863668 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d55b94afe9884d6d8b9f209875c7656d" } modify_external_catalogs: true
I20240507 08:12:10.863828 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a185e645f9d40f79750397df21fbad6" } modify_external_catalogs: true
W20240507 08:12:10.865319 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f748a59b4a347e288d534ef1296d72c consensus state has no leader
I20240507 08:12:10.865628 7100 consensus_queue.cc:1035] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.865669 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d75fc8323e274f5bb918bbcc918a336e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.866205 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d75fc8323e274f5bb918bbcc918a336e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:10.867234 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3769eb670ff442e2ad2fa5b6a934512a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3769eb670ff442e2ad2fa5b6a934512a already in progress: creating tablet
W20240507 08:12:10.869690 5251 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 117964us
I20240507 08:12:10.870391 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 97c978ec8bca4aa5b2ffaf22c3290747 (table default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]) successfully deleted
W20240507 08:12:10.870666 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 055a30bfc61b4ff2be9f2418ca59b030 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 055a30bfc61b4ff2be9f2418ca59b030 already in progress: creating tablet
W20240507 08:12:10.871065 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4ad95383487e458ca162038f6940271b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4ad95383487e458ca162038f6940271b already in progress: creating tablet
W20240507 08:12:10.871310 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b6ba3397e33b4cc7805a25e7535a383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6ba3397e33b4cc7805a25e7535a383a already in progress: creating tablet
I20240507 08:12:10.872085 7079 ts_tablet_manager.cc:1888] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.872320 7079 log.cc:1196] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4716c1285e91432a92b25ab13ada2779
I20240507 08:12:10.872726 7079 ts_tablet_manager.cc:1909] T 4716c1285e91432a92b25ab13ada2779 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.873952 7079 tablet_replica.cc:330] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.874217 7079 raft_consensus.cc:2229] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.874496 7079 pending_rounds.cc:62] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Trying to abort 1 pending ops.
I20240507 08:12:10.874608 7079 pending_rounds.cc:68] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924797646032896 op_type: NO_OP noop_request { }
I20240507 08:12:10.874799 7079 raft_consensus.cc:2866] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:10.874958 7079 raft_consensus.cc:2258] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:10.875080 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2bb648fe98e74248b5cd65603de58a02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2bb648fe98e74248b5cd65603de58a02 already in progress: creating tablet
W20240507 08:12:10.875367 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 93c68ddcfa7c40c180b6b12fabe165d6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93c68ddcfa7c40c180b6b12fabe165d6 already in progress: creating tablet
I20240507 08:12:10.876174 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (table default.Table_6fd5e754c95b480a8ecd9e7f593b7260 [id=0b044cc019d845b9a033666c78e88492]) successfully deleted
W20240507 08:12:10.885712 5654 consensus_peers.cc:485] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.887980 5840 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:10.888254 5840 ts_tablet_manager.cc:1367] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.000s
I20240507 08:12:10.873123 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6da541994d9c4b42b19de81f931ae930"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:10.889818 5839 tablet_bootstrap.cc:654] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.889998 7079 ts_tablet_manager.cc:1875] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:10.890125 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6da541994d9c4b42b19de81f931ae930 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:10.878137 5249 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 78195.4us
I20240507 08:12:10.892025 5843 raft_consensus.cc:374] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.892118 5843 raft_consensus.cc:729] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.892505 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:10.892844 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b4504b1ed3d449619926fe80469d9790 (table default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]) successfully deleted
I20240507 08:12:10.892882 5843 consensus_queue.cc:260] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:10.882586 5841 ts_tablet_manager.cc:1888] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240507 08:12:10.893112 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9ccbe2e368d14b56859d62d1cc742f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccbe2e368d14b56859d62d1cc742f90 already in progress: creating tablet
I20240507 08:12:10.893182 5841 log.cc:1196] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/0118ef28b24641d68cf24ce9df966243
I20240507 08:12:10.893301 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet dfa14901cdf14b72bea365b04ad521bd (table default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]) successfully deleted
I20240507 08:12:10.893451 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 33a425a90f0449bcaaacb4c3f653d26c (table default.Table_81657f34e12b4ca0b2fd8ee26ab0e663 [id=b1f2776824d24099bbc0110c3da97c2e]) successfully deleted
I20240507 08:12:10.893630 5841 ts_tablet_manager.cc:1909] T 0118ef28b24641d68cf24ce9df966243 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:10.893630 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4716c1285e91432a92b25ab13ada2779 (table default.Table_b3cc1770bc0f4c658f1ae0ef39b4aecb [id=bd83f9b19311401a8ff3fd39e8c75e8f]) successfully deleted
W20240507 08:12:10.878266 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a51f8aaf8b364d69b4f79ed4cceb5157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.889248 5840 raft_consensus.cc:348] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.897469 5840 raft_consensus.cc:374] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.897526 5840 raft_consensus.cc:729] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.897718 5840 consensus_queue.cc:260] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.885648 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c91f44a911c74b99ae52369cf8d517a3" } modify_external_catalogs: true
I20240507 08:12:10.900264 5843 ts_tablet_manager.cc:1398] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.045s user 0.001s sys 0.001s
I20240507 08:12:10.900364 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c91f44a911c74b99ae52369cf8d517a3" } modify_external_catalogs: true
I20240507 08:12:10.885823 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5e958b4da1dd43ec9cc03a7554860b1c" } modify_external_catalogs: true
I20240507 08:12:10.900831 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 793) took 5310 ms. Trace:
I20240507 08:12:10.885946 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d55b94afe9884d6d8b9f209875c7656d" } modify_external_catalogs: true
I20240507 08:12:10.900945 5841 rpcz_store.cc:276] 0507 08:12:05.590510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.590784 (+ 274us) service_pool.cc:225] Handling call
0507 08:12:10.713886 (+5123102us) ts_tablet_manager.cc:1120] Deleting tablet 0118ef28b24641d68cf24ce9df966243
0507 08:12:10.713895 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.779334 (+ 65439us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.882490 (+103156us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.900808 (+ 18318us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6347,"tablet-delete.queue_time_us":5122884}
I20240507 08:12:10.904703 6513 ts_tablet_manager.cc:1888] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.904254 7079 ts_tablet_manager.cc:1888] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.904934 6513 log.cc:1196] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/87377661d9374cef8ab1a97042b7f1b9
I20240507 08:12:10.905023 7079 log.cc:1196] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e555a89d340448bc852f2441ca850213
I20240507 08:12:10.905397 6513 ts_tablet_manager.cc:1909] T 87377661d9374cef8ab1a97042b7f1b9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.905460 7079 ts_tablet_manager.cc:1909] T e555a89d340448bc852f2441ca850213 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.905370 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_87151ec671d24d5d8ba8c66e0f7c6f85" } new_table_name: "default.Table_e22a29266ff54905b0152cf90c36bff0" modify_external_catalogs: true
I20240507 08:12:10.908388 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e555a89d340448bc852f2441ca850213 (table default.Table_948f06376d4849c79cc3f222a3fcaaad [id=a19f35cb006a4367961f26227bc0a662]) successfully deleted
I20240507 08:12:10.908701 7079 tablet_replica.cc:330] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.908931 7079 raft_consensus.cc:2229] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.910465 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16de5f84b4f3402ebb29a677269ab0a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.910992 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16de5f84b4f3402ebb29a677269ab0a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.886046 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a185e645f9d40f79750397df21fbad6" } modify_external_catalogs: true
I20240507 08:12:10.913352 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_91f1591433454159bdb582ad57a698d8" } new_table_name: "default.Table_0ecd39312f13432c947bfdf7fc8b7e7e" modify_external_catalogs: true
I20240507 08:12:10.914047 5843 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:10.914589 7079 raft_consensus.cc:2258] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.915167 7079 ts_tablet_manager.cc:1875] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.915879 5843 tablet_bootstrap.cc:654] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.917165 5839 tablet_bootstrap.cc:492] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:10.917373 5839 ts_tablet_manager.cc:1367] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20240507 08:12:10.917987 6934 raft_consensus.cc:482] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:10.918104 6934 raft_consensus.cc:504] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.918208 5839 raft_consensus.cc:348] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.918356 5839 raft_consensus.cc:374] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:10.918403 5839 raft_consensus.cc:729] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:10.918615 5839 consensus_queue.cc:260] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.918723 6934 leader_election.cc:288] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:10.920186 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:10.913785 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_faf3a930a35b4a80aa6be69297d5ed7b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.920894 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_faf3a930a35b4a80aa6be69297d5ed7b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:10.920941 5377 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.921167 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:10.921663 5378 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:10.921751 5378 leader_election.cc:302] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:10.915656 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b15c8ad2a8324c75a1ef49e219b962cf consensus state has no leader
I20240507 08:12:10.922124 6934 raft_consensus.cc:2726] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:10.919474 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_35152415e8264b5495c852cb2a68506c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.922508 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35152415e8264b5495c852cb2a68506c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.923187 5258 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
W20240507 08:12:10.925182 5251 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 49040.3us
I20240507 08:12:10.925523 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d451d03349794726a71c255a7f177436 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
W20240507 08:12:10.925812 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2b13fbddcab5460db69f43e63f490fba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b13fbddcab5460db69f43e63f490fba already in progress: creating tablet
I20240507 08:12:10.926082 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 0118ef28b24641d68cf24ce9df966243 (table default.Table_727fd7e435fd40368f63b993db76af91 [id=786a00558b0f42cd9700cd3200dfe1ac]) successfully deleted
I20240507 08:12:10.926915 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet aa094773f3bf418492bcd5b5021fd429 with delete_type TABLET_DATA_DELETED (replaced by tablet 451f566fdf2a40a7a61a2d7ae38fffba) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.927834 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.928313 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd with delete_type TABLET_DATA_DELETED (replaced by tablet c534708f1abd4a33965eea92bdc0d0de) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.928788 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:10.930155 5654 consensus_peers.cc:485] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:10.930389 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.930816 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.932060 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 820) took 4630 ms. Trace:
I20240507 08:12:10.932276 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7004d9f9040b4d57abb409a848b45e48" } new_table_name: "default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5" modify_external_catalogs: true
I20240507 08:12:10.932714 6513 rpcz_store.cc:276] 0507 08:12:06.301849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.344962 (+ 43113us) service_pool.cc:225] Handling call
0507 08:12:10.859948 (+4514986us) ts_tablet_manager.cc:1120] Deleting tablet 87377661d9374cef8ab1a97042b7f1b9
0507 08:12:10.859958 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.880450 (+ 20492us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.904608 (+ 24158us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.932041 (+ 27433us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25795,"tablet-delete.queue_time_us":4514853}
I20240507 08:12:10.933408 6513 tablet_replica.cc:330] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:10.933840 6513 raft_consensus.cc:2229] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.933650 5841 tablet_replica.cc:330] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:10.934114 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a16c963f4cf149fd9ff050aab7c68dbc" } modify_external_catalogs: true
I20240507 08:12:10.934296 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a16c963f4cf149fd9ff050aab7c68dbc" } modify_external_catalogs: true
I20240507 08:12:10.934430 6513 raft_consensus.cc:2258] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.934389 5841 raft_consensus.cc:2229] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:10.934721 5841 pending_rounds.cc:62] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:10.934818 5841 pending_rounds.cc:68] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924793962274816 op_type: NO_OP noop_request { }
I20240507 08:12:10.934971 5841 raft_consensus.cc:2866] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:10.934988 6513 ts_tablet_manager.cc:1875] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.935159 5841 raft_consensus.cc:2258] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.935822 5841 ts_tablet_manager.cc:1875] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.936112 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_27e9e4ffb901470db2653f82d93365ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.936587 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27e9e4ffb901470db2653f82d93365ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.938737 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 87377661d9374cef8ab1a97042b7f1b9 (table default.Table_ce35197b07dc454c82587e250579e404 [id=17853c4aa94a46138e3fdeaca0bc65e1]) successfully deleted
W20240507 08:12:10.940562 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44af2d81508a4e3caf774d3643d8136b consensus state has no leader
I20240507 08:12:10.940881 6508 consensus_queue.cc:1035] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:10.941283 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5392502ff2394c5d947c372642154554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.941728 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5392502ff2394c5d947c372642154554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.942934 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.943504 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.944082 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ccbe2e368d14b56859d62d1cc742f90 with delete_type TABLET_DATA_DELETED (replaced by tablet 096624b740fb4ac88a701a630c97b3eb) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:10.946344 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 2817163a934b4041bd139803ab6e0f8d (DEFAULT_TABLE table=default.Table_0f7492244d9a44b3998beb1d5a959eeb [id=383c9ba314894e2d97650da7327c6b03]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.946707 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2817163a934b4041bd139803ab6e0f8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.951768 6513 ts_tablet_manager.cc:1888] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.951886 6513 log.cc:1196] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b4504b1ed3d449619926fe80469d9790
I20240507 08:12:10.952243 6513 ts_tablet_manager.cc:1909] T b4504b1ed3d449619926fe80469d9790 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:10.953016 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 821) took 4651 ms. Trace:
I20240507 08:12:10.933432 5840 ts_tablet_manager.cc:1398] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.045s user 0.000s sys 0.002s
I20240507 08:12:10.953692 7079 ts_tablet_manager.cc:1888] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:10.953955 5840 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:10.953887 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ad968e67c91142a99564460bd9f033d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:10.954382 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad968e67c91142a99564460bd9f033d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:10.954787 7079 log.cc:1196] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/a7df45c39d1b490db97c72a0eb6cf3ba
I20240507 08:12:10.955935 5839 ts_tablet_manager.cc:1398] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.038s user 0.002s sys 0.000s
I20240507 08:12:10.957871 5839 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:10.959234 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:10.959978 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f33de0d21b1417680053925d89ecf91" } new_table_name: "default.Table_8d93f738313541cf822a03b4a819d93c" modify_external_catalogs: true
I20240507 08:12:10.971805 5843 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:10.973502 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7943787b81d45f08cada28f13694f66 consensus state has no leader
I20240507 08:12:10.976447 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 411619d6fb934f63b059ae7a489c8bf8 (DEFAULT_TABLE table=default.Table_a34af5c61e5649e5abc9c88235891674 [id=8b27bd9862864cafa09a94326eaa3218]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.978973 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 411619d6fb934f63b059ae7a489c8bf8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.980890 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 50358f99145c476fb3b0f8f5e2487aab (DEFAULT_TABLE table=default.Table_6991af2fca624791af1a67498a53b156 [id=910107a4e3854212a3cad0ddb4888b96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.981314 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50358f99145c476fb3b0f8f5e2487aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.977855 5413 tablet_service.cc:1467] Processing CreateTablet for tablet e241dd3c1af14b278b1632481edd0f30 (DEFAULT_TABLE table=default.Table_eb79af3080394424a223e89504916751 [id=7d909580fa6c4e23aecdfae42853ec86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.981638 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e241dd3c1af14b278b1632481edd0f30. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.982614 5839 tablet_bootstrap.cc:654] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.983021 7079 ts_tablet_manager.cc:1909] T a7df45c39d1b490db97c72a0eb6cf3ba P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.984599 7079 tablet_replica.cc:330] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:10.984787 7079 raft_consensus.cc:2229] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:10.984949 7079 raft_consensus.cc:2258] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:10.986075 7079 ts_tablet_manager.cc:1875] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:10.986989 5843 ts_tablet_manager.cc:1367] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.002s
I20240507 08:12:10.988050 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet a7df45c39d1b490db97c72a0eb6cf3ba (table default.Table_0fafe65af5f34be4b333cb6baa327f94 [id=58282ffddcba4e4db7736a325a4a530a]) successfully deleted
W20240507 08:12:10.988420 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6f8f4205f1f48abaf4fcd7113ad44a8 already in progress: creating tablet
W20240507 08:12:10.988780 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aa094773f3bf418492bcd5b5021fd429 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa094773f3bf418492bcd5b5021fd429 already in progress: creating tablet
I20240507 08:12:10.989964 5424 tablet_service.cc:1467] Processing CreateTablet for tablet a8adad098c934d11982d39aa1cf8b4bc (DEFAULT_TABLE table=default.Table_c39926d459d14c72884a82bf549db3ff [id=7c8f64d9bf2b48b5abba9f9ed53b31e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.990346 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8adad098c934d11982d39aa1cf8b4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.995299 7079 ts_tablet_manager.cc:1888] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:10.995240 6513 rpcz_store.cc:276] 0507 08:12:06.301958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.345107 (+ 43149us) service_pool.cc:225] Handling call
0507 08:12:10.933361 (+4588254us) ts_tablet_manager.cc:1120] Deleting tablet b4504b1ed3d449619926fe80469d9790
0507 08:12:10.933371 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.947248 (+ 13877us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.951687 (+ 4439us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:10.952995 (+ 1308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4588113}
I20240507 08:12:10.996167 7079 log.cc:1196] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/9e447562e82347a584e39586500c27c2
I20240507 08:12:10.996554 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b4504b1ed3d449619926fe80469d9790 (table default.Table_420e2e62b141454298f31e925f51f2d8 [id=02d7867fd75e4b22915a7a20a10378f9]) successfully deleted
I20240507 08:12:10.996647 7079 ts_tablet_manager.cc:1909] T 9e447562e82347a584e39586500c27c2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:10.998183 5422 tablet_service.cc:1467] Processing CreateTablet for tablet b7943787b81d45f08cada28f13694f66 (DEFAULT_TABLE table=default.Table_0f33de0d21b1417680053925d89ecf91 [id=2724329ee1f1487bb0ff3b9aaab9030a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.998534 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 5934f03ca4a949b79998b8606b92f440 (DEFAULT_TABLE table=default.Table_5bc2fbd1f2a144369c08d69027aaf373 [id=794b5a14afe14608b5e50295fbc585f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:10.998661 5840 tablet_bootstrap.cc:654] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:10.998548 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7943787b81d45f08cada28f13694f66. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:10.999037 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 9e447562e82347a584e39586500c27c2 (table default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]) successfully deleted
I20240507 08:12:10.993470 5843 raft_consensus.cc:348] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:10.999714 5843 raft_consensus.cc:374] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:10.999735 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 81fcf82e60c94e9eaadd75bd37595261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81fcf82e60c94e9eaadd75bd37595261 already in progress: creating tablet
I20240507 08:12:10.999790 5843 raft_consensus.cc:729] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.000015 5843 consensus_queue.cc:260] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.000310 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 822) took 4682 ms. Trace:
I20240507 08:12:11.000393 6513 rpcz_store.cc:276] 0507 08:12:06.318063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.345258 (+ 27195us) service_pool.cc:225] Handling call
0507 08:12:11.000181 (+4654923us) ts_tablet_manager.cc:1120] Deleting tablet e5c1bc3385134d8fbcc94458ef0612ec
0507 08:12:11.000190 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.000291 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4654803}
W20240507 08:12:11.000802 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e5c1bc3385134d8fbcc94458ef0612ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c1bc3385134d8fbcc94458ef0612ec already in progress: creating tablet
I20240507 08:12:11.001631 5843 ts_tablet_manager.cc:1398] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240507 08:12:11.002069 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5934f03ca4a949b79998b8606b92f440. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.002269 6513 tablet_replica.cc:330] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.002452 6513 raft_consensus.cc:2229] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.002801 6513 pending_rounds.cc:62] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:11.002854 6513 pending_rounds.cc:68] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924781935448064 op_type: NO_OP noop_request { }
I20240507 08:12:11.002945 6513 raft_consensus.cc:2866] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:11.003008 6513 raft_consensus.cc:2258] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:11.003144 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 39e5e67df91f4029bf55e5a82dc31f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39e5e67df91f4029bf55e5a82dc31f82 already in progress: creating tablet
I20240507 08:12:11.004367 5843 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:11.006096 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 128c2034d917473eb5c496319fdc9625 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c2034d917473eb5c496319fdc9625 already in progress: creating tablet
W20240507 08:12:11.006381 5377 consensus_peers.cc:485] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 077ccc3946b34008bea7b80d8aee2662. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:11.006858 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 924d8d2fe2f148cba13f107e4c70fc68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924d8d2fe2f148cba13f107e4c70fc68 already in progress: creating tablet
I20240507 08:12:11.003528 6513 ts_tablet_manager.cc:1875] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.007308 7079 tablet_replica.cc:330] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.007472 7079 raft_consensus.cc:2229] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.007606 7079 raft_consensus.cc:2258] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.008602 5843 tablet_bootstrap.cc:654] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.010164 5841 ts_tablet_manager.cc:1888] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.010591 5841 log.cc:1196] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:11.011216 5841 ts_tablet_manager.cc:1909] T 1037c3c0af5e464bb23a311e552fcf21 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.016718 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 794) took 5426 ms. Trace:
I20240507 08:12:11.018467 5841 rpcz_store.cc:276] 0507 08:12:05.590668 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.591030 (+ 362us) service_pool.cc:225] Handling call
0507 08:12:10.933588 (+5342558us) ts_tablet_manager.cc:1120] Deleting tablet 1037c3c0af5e464bb23a311e552fcf21
0507 08:12:10.933601 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:10.995009 (+ 61408us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.010058 (+ 15049us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.016700 (+ 6642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5342370}
I20240507 08:12:11.019871 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 795) took 5331 ms. Trace:
I20240507 08:12:11.020532 5841 rpcz_store.cc:276] 0507 08:12:05.688551 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.688609 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:11.019746 (+5331137us) ts_tablet_manager.cc:1120] Deleting tablet c9e2deaadfe34c2dbdc9819729cb4890
0507 08:12:11.019757 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.019856 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5330896}
I20240507 08:12:11.021656 5841 tablet_replica.cc:330] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.022485 5841 raft_consensus.cc:2229] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.023484 5841 raft_consensus.cc:2258] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.024885 5841 ts_tablet_manager.cc:1875] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:11.025487 5250 consensus_peers.cc:485] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d -> Peer a39de2a69ed4492aab4d7a232b09c1ae (127.4.246.253:43459): Couldn't send request to peer a39de2a69ed4492aab4d7a232b09c1ae. Status: Network error: Client connection negotiation failed: client connection to 127.4.246.253:43459: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240507 08:12:11.031225 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 924d8d2fe2f148cba13f107e4c70fc68 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.031561 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 128c2034d917473eb5c496319fdc9625 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.031757 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c49745fce2949d1909baf7583b157a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.032035 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.032296 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e5e67df91f4029bf55e5a82dc31f82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.034701 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1037c3c0af5e464bb23a311e552fcf21 (table default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]) successfully deleted
W20240507 08:12:11.035149 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c9e2deaadfe34c2dbdc9819729cb4890 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9e2deaadfe34c2dbdc9819729cb4890 already in progress: creating tablet
I20240507 08:12:11.035432 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 81fcf82e60c94e9eaadd75bd37595261 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.039798 6675 consensus_queue.cc:1035] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:11.043707 5654 consensus_peers.cc:485] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.046815 5841 ts_tablet_manager.cc:1888] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20240507 08:12:11.059978 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet a7df45c39d1b490db97c72a0eb6cf3ba from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.060426 6513 ts_tablet_manager.cc:1888] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.061807 6513 log.cc:1196] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/168e49ae24314a7aae2291485ddeda44
I20240507 08:12:11.062184 6513 ts_tablet_manager.cc:1909] T 168e49ae24314a7aae2291485ddeda44 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.062978 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 817) took 4777 ms. Trace:
I20240507 08:12:11.063059 6513 rpcz_store.cc:276] 0507 08:12:06.285628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.340826 (+ 55198us) service_pool.cc:225] Handling call
0507 08:12:11.002220 (+4661394us) ts_tablet_manager.cc:1120] Deleting tablet 168e49ae24314a7aae2291485ddeda44
0507 08:12:11.002232 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.019436 (+ 17204us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.060299 (+ 40863us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.062958 (+ 2659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641249}
I20240507 08:12:11.063355 6513 tablet_replica.cc:330] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.063529 6513 raft_consensus.cc:2229] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.063678 6513 raft_consensus.cc:2258] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.064267 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 168e49ae24314a7aae2291485ddeda44 (table default.Table_c58677b45a2b461490f3de5ca89c7b21 [id=be2e00744c2548a898ff49b4ee6ce710]) successfully deleted
I20240507 08:12:11.066318 7079 ts_tablet_manager.cc:1875] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.066644 5840 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.066797 5840 ts_tablet_manager.cc:1367] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.007s
I20240507 08:12:11.067653 5840 raft_consensus.cc:348] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.067898 5840 raft_consensus.cc:374] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.067962 5840 raft_consensus.cc:729] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.068838 5841 log.cc:1196] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c1c6b488b9014841899597d858bda5e2
I20240507 08:12:11.074424 7176 tablet.cc:1722] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:11.076159 5841 ts_tablet_manager.cc:1909] T c1c6b488b9014841899597d858bda5e2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.078752 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 796) took 5390 ms. Trace:
I20240507 08:12:11.079732 7044 raft_consensus.cc:482] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.079874 7044 raft_consensus.cc:504] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.080471 7044 leader_election.cc:288] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.081755 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7841f5ebf5ed4f6eb28dcbd4087f1713" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:11.081844 5520 leader_election.cc:341] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:12:11.082360 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b98946e635c74c888d87a7ed5a3437c2" } new_table_name: "default.Table_446a41da25a64da8969bccf51fab7fa5" modify_external_catalogs: true
I20240507 08:12:11.082859 5517 leader_election.cc:302] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:11.083276 7044 raft_consensus.cc:2726] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.083312 5841 rpcz_store.cc:276] 0507 08:12:05.688673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.689693 (+ 1020us) service_pool.cc:225] Handling call
0507 08:12:11.021557 (+5331864us) ts_tablet_manager.cc:1120] Deleting tablet c1c6b488b9014841899597d858bda5e2
0507 08:12:11.021604 (+ 47us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.036375 (+ 14771us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.046721 (+ 10346us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.078726 (+ 32005us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5331613}
I20240507 08:12:11.083801 6675 raft_consensus.cc:482] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.083918 6675 raft_consensus.cc:504] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.084514 6675 leader_election.cc:288] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.084856 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.085021 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.085822 5443 raft_consensus.cc:2454] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.086675 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c1c6b488b9014841899597d858bda5e2 (table default.Table_48b51787d9ce4ff1b9b8744ed86885bb [id=0a1bc06a351549d5aea075ad8d026476]) successfully deleted
W20240507 08:12:11.085394 5654 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.087164 5656 leader_election.cc:302] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.087646 6675 raft_consensus.cc:2781] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.085834 5841 tablet_replica.cc:330] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.087731 6675 raft_consensus.cc:482] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.087805 6675 raft_consensus.cc:3037] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.087898 5841 raft_consensus.cc:2229] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.088651 5843 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.088768 5843 ts_tablet_manager.cc:1367] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.084s user 0.007s sys 0.009s
I20240507 08:12:11.089821 5841 raft_consensus.cc:2258] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.090332 5841 ts_tablet_manager.cc:1875] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:11.090636 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c92727908174c4d9f6aa659569027ff has no consensus state
I20240507 08:12:11.090518 6675 raft_consensus.cc:504] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.091300 5843 raft_consensus.cc:348] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.091490 5843 raft_consensus.cc:374] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.092080 6675 leader_election.cc:288] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.091913 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.092203 5443 raft_consensus.cc:3037] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.092793 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "150efae7834740b798af86ae1dcfe445" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:11.093254 5654 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.094982 5443 raft_consensus.cc:2454] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.095348 7177 tablet.cc:1722] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:11.095655 5656 leader_election.cc:302] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.096112 6675 raft_consensus.cc:2781] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.096302 6675 raft_consensus.cc:686] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.096694 6675 consensus_queue.cc:237] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.097939 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0c51ec8f693e4f108100c75470ead2a9
I20240507 08:12:11.098737 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.100319 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.100615 5843 raft_consensus.cc:729] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.101545 5843 consensus_queue.cc:260] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.102437 5841 ts_tablet_manager.cc:1888] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:11.102599 5841 log.cc:1196] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/077ccc3946b34008bea7b80d8aee2662
I20240507 08:12:11.103055 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7fbe0f58bad541e7bcfb611105328d56
I20240507 08:12:11.104254 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.104331 5841 ts_tablet_manager.cc:1909] T 077ccc3946b34008bea7b80d8aee2662 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.104846 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.105819 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 797) took 5417 ms. Trace:
I20240507 08:12:11.105952 5841 rpcz_store.cc:276] 0507 08:12:05.688788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.689978 (+ 1190us) service_pool.cc:225] Handling call
0507 08:12:11.085777 (+5395799us) ts_tablet_manager.cc:1120] Deleting tablet 077ccc3946b34008bea7b80d8aee2662
0507 08:12:11.085788 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.095741 (+ 9953us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.102323 (+ 6582us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.105799 (+ 3476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5395602}
I20240507 08:12:11.106453 5839 tablet_bootstrap.cc:492] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.106439 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 803) took 5223 ms. Trace:
I20240507 08:12:11.106580 5839 ts_tablet_manager.cc:1367] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.149s user 0.026s sys 0.023s
I20240507 08:12:11.106514 5841 rpcz_store.cc:276] 0507 08:12:05.883262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:05.883954 (+ 692us) service_pool.cc:225] Handling call
0507 08:12:11.106306 (+5222352us) ts_tablet_manager.cc:1120] Deleting tablet 3618b416194f4effb29b823709593359
0507 08:12:11.106315 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.106419 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5222092}
I20240507 08:12:11.106813 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 826) took 5096 ms. Trace:
I20240507 08:12:11.106876 5841 rpcz_store.cc:276] 0507 08:12:06.009978 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.162148 (+152170us) service_pool.cc:225] Handling call
0507 08:12:11.106709 (+4944561us) ts_tablet_manager.cc:1120] Deleting tablet 12e85df8b900468ba2584fd8f2e9bb2d
0507 08:12:11.106716 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.106800 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944368}
I20240507 08:12:11.107177 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 827) took 5016 ms. Trace:
I20240507 08:12:11.107247 5841 rpcz_store.cc:276] 0507 08:12:06.091086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.162352 (+ 71266us) service_pool.cc:225] Handling call
0507 08:12:11.107091 (+4944739us) ts_tablet_manager.cc:1120] Deleting tablet c7d19b7cc39a4337826819a20205aecc
0507 08:12:11.107099 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.107164 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944615}
I20240507 08:12:11.107331 5839 raft_consensus.cc:348] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.107477 5841 tablet_replica.cc:330] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.107499 5839 raft_consensus.cc:374] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.107546 5839 raft_consensus.cc:729] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.107647 5841 raft_consensus.cc:2229] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.107779 5841 raft_consensus.cc:2258] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.107729 5839 consensus_queue.cc:260] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.108217 5839 ts_tablet_manager.cc:1398] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:11.108290 5841 ts_tablet_manager.cc:1875] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.108539 5839 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.109700 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fbe0f58bad541e7bcfb611105328d56 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.110423 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 077ccc3946b34008bea7b80d8aee2662 (table default.Table_bf6b95a29b9d466eabf2276567fe5fa9 [id=34d5ccf8d18945608504d42ba58ad664]) successfully deleted
I20240507 08:12:11.112455 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1633148e0c444d8eaedb7e613aa6ed3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.113126 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1633148e0c444d8eaedb7e613aa6ed3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.113445 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e63c3f3ecc84913ad05864be450e9b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 279e95d9804c44eaa2c23a4ec02c84ff) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.113634 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d3257e1057da421d901ffdb965228250"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.114081 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3257e1057da421d901ffdb965228250 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.110869 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c51ec8f693e4f108100c75470ead2a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:11.114972 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3618b416194f4effb29b823709593359 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3618b416194f4effb29b823709593359 already in progress: creating tablet
W20240507 08:12:11.115267 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 12e85df8b900468ba2584fd8f2e9bb2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12e85df8b900468ba2584fd8f2e9bb2d already in progress: creating tablet
W20240507 08:12:11.115552 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c7d19b7cc39a4337826819a20205aecc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7d19b7cc39a4337826819a20205aecc already in progress: creating tablet
I20240507 08:12:11.118011 5843 ts_tablet_manager.cc:1398] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20240507 08:12:11.119522 7079 ts_tablet_manager.cc:1888] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.120478 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2c3c93e036544172b7ddb34378d99f86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.120985 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c3c93e036544172b7ddb34378d99f86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.119258 5843 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.122644 5843 tablet_bootstrap.cc:654] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.124166 6880 raft_consensus.cc:482] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.124286 6880 raft_consensus.cc:504] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.124868 6880 leader_election.cc:288] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.125339 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_63a2b5e8edd14d6aa283025fbf8df787"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.125875 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_63a2b5e8edd14d6aa283025fbf8df787 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.126243 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.126511 5716 raft_consensus.cc:2454] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:11.125474 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.127471 5377 leader_election.cc:302] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.127919 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_46792e71ef0e46df88355bf7c54afbd1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.128373 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46792e71ef0e46df88355bf7c54afbd1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:11.129156 5378 leader_election.cc:341] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.130235 6880 raft_consensus.cc:2781] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.130312 6880 raft_consensus.cc:482] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.130230 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1971f4cb4ebc464586d955900fb585ed" } new_table_name: "default.Table_951d582b8481445ca1e0cd6d9e718595" modify_external_catalogs: true
I20240507 08:12:11.130374 6880 raft_consensus.cc:3037] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.120101 7079 log.cc:1196] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/302fbf120e9d472ba6bdf9e6cbdf6dbd
I20240507 08:12:11.134500 7079 ts_tablet_manager.cc:1909] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:11.134589 5840 consensus_queue.cc:260] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.135288 5840 ts_tablet_manager.cc:1398] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20240507 08:12:11.135787 6513 ts_tablet_manager.cc:1875] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.136124 7055 tablet.cc:1722] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:11.135886 6880 raft_consensus.cc:504] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.136922 5843 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.137354 6880 leader_election.cc:288] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.138453 5843 ts_tablet_manager.cc:1367] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240507 08:12:11.140938 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:11.141134 5716 raft_consensus.cc:3037] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.141327 5839 tablet_bootstrap.cc:654] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.143574 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.140691 5843 raft_consensus.cc:348] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.145365 5843 raft_consensus.cc:374] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.145458 5716 raft_consensus.cc:2454] T eb1901ddec07495abba085e7188c00d0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:11.145697 5843 raft_consensus.cc:729] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.145922 5377 leader_election.cc:302] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.146195 6880 raft_consensus.cc:2781] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.146238 5843 consensus_queue.cc:260] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.146409 6880 raft_consensus.cc:686] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:11.146646 6880 consensus_queue.cc:237] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.147675 5843 ts_tablet_manager.cc:1398] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:11.147917 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_56208b251bcf49dc8b79aac94c9ca7b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.148483 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56208b251bcf49dc8b79aac94c9ca7b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.149183 5843 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.149199 5840 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.149984 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c75f62ac2a774ef79d3238ef5ff7d317"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.150629 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c75f62ac2a774ef79d3238ef5ff7d317 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.150840 5840 tablet_bootstrap.cc:654] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.151007 5843 tablet_bootstrap.cc:654] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.156548 7044 raft_consensus.cc:482] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.156715 7044 raft_consensus.cc:504] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.157476 7044 leader_election.cc:288] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:11.159047 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 665b3736350a4121ac62014a81e5669d consensus state has no leader
I20240507 08:12:11.159329 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.159442 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0ecd39312f13432c947bfdf7fc8b7e7e" } new_table_name: "default.Table_4ef0b0afa8c44d09b17db123d8333e0c" modify_external_catalogs: true
I20240507 08:12:11.159868 5583 raft_consensus.cc:3037] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.161626 6784 raft_consensus.cc:482] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.161787 6784 raft_consensus.cc:504] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.162102 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8d93f738313541cf822a03b4a819d93c" } new_table_name: "default.Table_fe03d3533c314ba7a8d576b4c4055336" modify_external_catalogs: true
I20240507 08:12:11.162334 6784 leader_election.cc:288] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.163085 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.163725 5654 leader_election.cc:341] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.165128 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (table default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]) successfully deleted
W20240507 08:12:11.165423 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5c49745fce2949d1909baf7583b157a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c49745fce2949d1909baf7583b157a7 already in progress: creating tablet
I20240507 08:12:11.165935 7079 tablet_replica.cc:330] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.166203 7079 raft_consensus.cc:2229] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.166429 7079 raft_consensus.cc:2258] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.158179 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89db2847c09b4c74b809bd13f78d0b8b" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:11.158550 5378 leader_election.cc:341] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.167382 6513 ts_tablet_manager.cc:1888] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.167505 6513 log.cc:1196] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/97c978ec8bca4aa5b2ffaf22c3290747
I20240507 08:12:11.164448 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.167699 5445 raft_consensus.cc:2454] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.168262 7079 ts_tablet_manager.cc:1875] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.168483 5656 leader_election.cc:302] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.170329 6513 ts_tablet_manager.cc:1909] T 97c978ec8bca4aa5b2ffaf22c3290747 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.171679 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 818) took 4870 ms. Trace:
I20240507 08:12:11.172621 6513 rpcz_store.cc:276] 0507 08:12:06.301618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.341002 (+ 39384us) service_pool.cc:225] Handling call
0507 08:12:11.063311 (+4722309us) ts_tablet_manager.cc:1120] Deleting tablet 97c978ec8bca4aa5b2ffaf22c3290747
0507 08:12:11.063320 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.153317 (+ 89997us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.167259 (+ 13942us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.171636 (+ 4377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4681092}
I20240507 08:12:11.174225 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 97c978ec8bca4aa5b2ffaf22c3290747 (table default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]) successfully deleted
I20240507 08:12:11.172217 6675 raft_consensus.cc:2781] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.173795 6784 raft_consensus.cc:482] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.168234 5520 leader_election.cc:302] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:11.174686 6784 raft_consensus.cc:504] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.175007 7044 raft_consensus.cc:2726] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.175469 6784 leader_election.cc:288] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.176225 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.176488 5445 raft_consensus.cc:2454] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.176882 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_695a3d2597174aa7bc8a61e91f85f9a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:11.177017 5656 leader_election.cc:302] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:11.177558 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_695a3d2597174aa7bc8a61e91f85f9a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.177791 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.178437 5654 leader_election.cc:341] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.178819 6784 raft_consensus.cc:2781] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.178920 6784 raft_consensus.cc:482] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.179013 6784 raft_consensus.cc:3037] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.182272 6784 raft_consensus.cc:504] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.182495 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2a9bf32660374d72a70a5c5b228881a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.183142 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a9bf32660374d72a70a5c5b228881a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.183863 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:11.184573 5654 leader_election.cc:341] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.185606 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44af2d81508a4e3caf774d3643d8136b consensus state has no leader
I20240507 08:12:11.186074 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.186309 5445 raft_consensus.cc:3037] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.189873 6784 leader_election.cc:288] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.190021 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b2afebeaafe84471bf8b1ae310b46036" } new_table_name: "default.Table_5fb384ed8b7546e39c39acc6facec9f5" modify_external_catalogs: true
I20240507 08:12:11.190191 5445 raft_consensus.cc:2454] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.190668 5656 leader_election.cc:302] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.191228 6784 raft_consensus.cc:2781] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.191865 6784 raft_consensus.cc:686] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:11.192065 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b7943787b81d45f08cada28f13694f66 consensus state has no leader
I20240507 08:12:11.192261 6784 consensus_queue.cc:237] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.192610 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9e717d2407ba4a54af3626573fb38c0b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.193183 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e717d2407ba4a54af3626573fb38c0b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.193491 6675 raft_consensus.cc:482] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.193706 6675 raft_consensus.cc:3037] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.194097 5841 ts_tablet_manager.cc:1888] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.194260 5840 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.194404 5840 ts_tablet_manager.cc:1367] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.000s
I20240507 08:12:11.195134 5840 raft_consensus.cc:348] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.195328 5840 raft_consensus.cc:374] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.195474 5840 raft_consensus.cc:729] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.196146 5840 consensus_queue.cc:260] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.196357 6675 raft_consensus.cc:504] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.201365 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.201694 5841 log.cc:1196] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/9ced879f35e14106b2609c78b7f24002
W20240507 08:12:11.202009 5654 leader_election.cc:341] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.202104 5841 ts_tablet_manager.cc:1909] T 9ced879f35e14106b2609c78b7f24002 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.198335 5840 ts_tablet_manager.cc:1398] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:11.203055 5840 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.200486 6513 tablet_replica.cc:330] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.204522 5840 tablet_bootstrap.cc:654] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.207821 6513 raft_consensus.cc:2229] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.206936 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 828) took 5083 ms. Trace:
I20240507 08:12:11.208056 5841 rpcz_store.cc:276] 0507 08:12:06.123327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.162481 (+ 39154us) service_pool.cc:225] Handling call
0507 08:12:11.107439 (+4944958us) ts_tablet_manager.cc:1120] Deleting tablet 9ced879f35e14106b2609c78b7f24002
0507 08:12:11.107447 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.153238 (+ 45791us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.193991 (+ 40753us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.206825 (+ 12834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4944812}
I20240507 08:12:11.208788 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 9ced879f35e14106b2609c78b7f24002 (table default.Table_dea0e2b1b090454cbd15a00f7e792d6e [id=aa09d786696945d5b4060819737a6e11]) successfully deleted
I20240507 08:12:11.209237 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 830) took 4960 ms. Trace:
I20240507 08:12:11.210180 5841 rpcz_store.cc:276] 0507 08:12:06.248593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.253600 (+ 5007us) service_pool.cc:225] Handling call
0507 08:12:11.209087 (+4955487us) ts_tablet_manager.cc:1120] Deleting tablet d33a3ccd7a5e4b9e956735443d4c484a
0507 08:12:11.209097 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.209197 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4955300}
W20240507 08:12:11.210700 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d33a3ccd7a5e4b9e956735443d4c484a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d33a3ccd7a5e4b9e956735443d4c484a already in progress: creating tablet
I20240507 08:12:11.211143 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 829) took 4962 ms. Trace:
I20240507 08:12:11.212517 5841 rpcz_store.cc:276] 0507 08:12:06.248477 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.257258 (+ 8781us) service_pool.cc:225] Handling call
0507 08:12:11.210982 (+4953724us) ts_tablet_manager.cc:1120] Deleting tablet 6def4bf6a7554a4090a39b1823c4097c
0507 08:12:11.210991 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.211105 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4953520}
W20240507 08:12:11.212994 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6def4bf6a7554a4090a39b1823c4097c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6def4bf6a7554a4090a39b1823c4097c already in progress: creating tablet
I20240507 08:12:11.213395 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 831) took 4893 ms. Trace:
I20240507 08:12:11.214219 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.214965 5841 rpcz_store.cc:276] 0507 08:12:06.320225 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.364545 (+ 44320us) service_pool.cc:225] Handling call
0507 08:12:11.213255 (+4848710us) ts_tablet_manager.cc:1120] Deleting tablet e5c1bc3385134d8fbcc94458ef0612ec
0507 08:12:11.213266 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.213349 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4848296}
I20240507 08:12:11.215132 5445 raft_consensus.cc:3037] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:11.215364 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e5c1bc3385134d8fbcc94458ef0612ec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c1bc3385134d8fbcc94458ef0612ec already in progress: creating tablet
I20240507 08:12:11.216104 5841 tablet_replica.cc:330] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.216329 5841 raft_consensus.cc:2229] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.216517 5841 raft_consensus.cc:2258] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.216974 7079 ts_tablet_manager.cc:1888] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:11.217090 7079 log.cc:1196] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:12:11.217590 7079 ts_tablet_manager.cc:1909] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:11.218740 5445 raft_consensus.cc:2454] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.219002 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:11.219089 5656 leader_election.cc:302] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.219466 6784 raft_consensus.cc:2781] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.219928 6784 raft_consensus.cc:686] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.220199 6784 consensus_queue.cc:237] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.221289 6513 pending_rounds.cc:62] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:11.221386 6513 pending_rounds.cc:68] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924796991291392 op_type: NO_OP noop_request { }
I20240507 08:12:11.221539 6513 raft_consensus.cc:2866] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:11.221698 6513 raft_consensus.cc:2258] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.222469 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (table default.Table_6fd5e754c95b480a8ecd9e7f593b7260 [id=0b044cc019d845b9a033666c78e88492]) successfully deleted
I20240507 08:12:11.219393 5839 tablet_bootstrap.cc:492] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.222790 6513 ts_tablet_manager.cc:1875] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.222808 5839 ts_tablet_manager.cc:1367] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.000s
I20240507 08:12:11.223567 5839 raft_consensus.cc:348] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.223842 5839 raft_consensus.cc:374] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.223965 5839 raft_consensus.cc:729] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.224138 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1] alter complete (version 1)
I20240507 08:12:11.224243 5839 consensus_queue.cc:260] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.225114 6675 leader_election.cc:288] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.228384 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ed2604a090c94c78bd4f7a1dea6f81b7 (table default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316]). Expected version 1 got 0
I20240507 08:12:11.230316 5843 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.230527 5843 ts_tablet_manager.cc:1367] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.009s
I20240507 08:12:11.232095 5843 raft_consensus.cc:348] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.232498 5839 ts_tablet_manager.cc:1398] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:11.232555 5843 raft_consensus.cc:374] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.232776 5843 raft_consensus.cc:729] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.233175 5843 consensus_queue.cc:260] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.234565 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.234709 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.234797 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.234907 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 168e49ae24314a7aae2291485ddeda44 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.235071 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet b7947ca2d8324dbb9c909ec50852fed8 with delete_type TABLET_DATA_DELETED (replaced by tablet 5f0b399a2d074f4ca3626ff17bd5518e) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.235416 5843 ts_tablet_manager.cc:1398] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:11.240474 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 665b3736350a4121ac62014a81e5669d (DEFAULT_TABLE table=default.Table_951d582b8481445ca1e0cd6d9e718595 [id=d8c6dc565a824968b1904345750faaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.240895 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b3736350a4121ac62014a81e5669d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.245244 5264 catalog_manager.cc:5484] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:11.245693 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:11.246129 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:11.248075 6513 ts_tablet_manager.cc:1888] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.248324 6513 log.cc:1196] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f294baf04bf74dfe8500cb8c1452d8b3
I20240507 08:12:11.248867 6513 ts_tablet_manager.cc:1909] T f294baf04bf74dfe8500cb8c1452d8b3 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.248867 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.250252 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 823) took 4690 ms. Trace:
I20240507 08:12:11.251184 5843 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:11.252352 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3535f1fbfe34f6e95e9895771ae1350 consensus state has no leader
I20240507 08:12:11.252914 5843 tablet_bootstrap.cc:654] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.255940 5841 ts_tablet_manager.cc:1875] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.258157 6513 rpcz_store.cc:276] 0507 08:12:06.559548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.559634 (+ 86us) service_pool.cc:225] Handling call
0507 08:12:11.200410 (+4640776us) ts_tablet_manager.cc:1120] Deleting tablet f294baf04bf74dfe8500cb8c1452d8b3
0507 08:12:11.200429 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.228871 (+ 28442us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.247980 (+ 19109us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.250234 (+ 2254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4640541}
I20240507 08:12:11.258867 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f294baf04bf74dfe8500cb8c1452d8b3 (table default.Table_6d8668c1f9b94e15b7b7603a49f11a7d [id=6f1cedff6040494bb0a9bfe571648812]) successfully deleted
I20240507 08:12:11.259361 6513 tablet_replica.cc:330] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.261018 6513 raft_consensus.cc:2229] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.261315 6513 pending_rounds.cc:62] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:11.261431 6513 pending_rounds.cc:68] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924799374594048 op_type: NO_OP noop_request { }
I20240507 08:12:11.261519 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 665b3736350a4121ac62014a81e5669d (DEFAULT_TABLE table=default.Table_951d582b8481445ca1e0cd6d9e718595 [id=d8c6dc565a824968b1904345750faaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.261651 6513 raft_consensus.cc:2866] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:11.261751 6513 raft_consensus.cc:2258] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.261937 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b3736350a4121ac62014a81e5669d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.262720 6513 ts_tablet_manager.cc:1875] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.265740 5839 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.261389 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 665b3736350a4121ac62014a81e5669d (DEFAULT_TABLE table=default.Table_951d582b8481445ca1e0cd6d9e718595 [id=d8c6dc565a824968b1904345750faaf5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.266254 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 665b3736350a4121ac62014a81e5669d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.268536 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet b6b9232426b948ae8634f7afd65df393 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.269716 6784 consensus_queue.cc:1035] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:11.272249 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0c51ec8f693e4f108100c75470ead2a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c51ec8f693e4f108100c75470ead2a9 already in progress: creating tablet
W20240507 08:12:11.274261 5654 consensus_peers.cc:485] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.274554 7079 tablet_replica.cc:330] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.274756 7079 raft_consensus.cc:2229] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.275146 5843 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.275295 5843 ts_tablet_manager.cc:1367] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.007s
I20240507 08:12:11.276055 5843 raft_consensus.cc:348] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.276268 5843 raft_consensus.cc:374] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:11.272876 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7fbe0f58bad541e7bcfb611105328d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbe0f58bad541e7bcfb611105328d56 already in progress: creating tablet
I20240507 08:12:11.276340 5843 raft_consensus.cc:729] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.276571 5843 consensus_queue.cc:260] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.276839 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.277204 5843 ts_tablet_manager.cc:1398] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:11.277225 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26c7ef6f2845cabe09d09045d4feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.277491 5843 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.277750 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.278110 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26c7ef6f2845cabe09d09045d4feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.283655 5839 tablet_bootstrap.cc:654] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.283862 7079 raft_consensus.cc:2258] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.286451 5843 tablet_bootstrap.cc:654] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.287235 7079 ts_tablet_manager.cc:1875] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.292834 7079 ts_tablet_manager.cc:1888] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.295419 5841 ts_tablet_manager.cc:1888] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.295888 6784 raft_consensus.cc:482] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.295979 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fdd7bcf579774a45a7fe204ceec43195" } new_table_name: "default.Table_6a769d7d87b84f1da5328510e326530f" modify_external_catalogs: true
I20240507 08:12:11.296032 6784 raft_consensus.cc:504] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.296700 6784 leader_election.cc:288] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.297793 5841 log.cc:1196] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7b4b84ab3f274185a2b838ebb969ab52
I20240507 08:12:11.297832 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.299176 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69142a1181436abc438211e681c73d (DEFAULT_TABLE table=default.Table_62410585affa4ee1aab69399909e9ff6 [id=a1448cdc66904f45ab32beeb47864e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.298951 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.299424 5443 raft_consensus.cc:2454] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.299556 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69142a1181436abc438211e681c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.299927 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e26c7ef6f2845cabe09d09045d4feda. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.299962 5656 leader_election.cc:302] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.300197 6784 raft_consensus.cc:2781] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.300256 6784 raft_consensus.cc:482] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.300304 6784 raft_consensus.cc:3037] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.302584 6784 raft_consensus.cc:504] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.303174 6784 leader_election.cc:288] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.304579 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.304774 5443 raft_consensus.cc:3037] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.305053 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6cdc86ef760942ebbb3794342c9ccac7" } modify_external_catalogs: true
I20240507 08:12:11.305254 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6cdc86ef760942ebbb3794342c9ccac7" } modify_external_catalogs: true
I20240507 08:12:11.307615 5443 raft_consensus.cc:2454] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.308075 5656 leader_election.cc:302] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.308360 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e80ae4bf74945cc82095e3a11e1939b" } modify_external_catalogs: true
I20240507 08:12:11.308531 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e80ae4bf74945cc82095e3a11e1939b" } modify_external_catalogs: true
I20240507 08:12:11.308684 6675 raft_consensus.cc:2781] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.308799 6675 raft_consensus.cc:686] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.309023 6675 consensus_queue.cc:237] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.312806 6513 ts_tablet_manager.cc:1888] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.312951 6513 log.cc:1196] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/150efae7834740b798af86ae1dcfe445
I20240507 08:12:11.313334 6513 ts_tablet_manager.cc:1909] T 150efae7834740b798af86ae1dcfe445 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.314229 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 824) took 4607 ms. Trace:
I20240507 08:12:11.315815 7182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.246.253:43459 every 8 connection(s)
I20240507 08:12:11.316546 6604 rpc_server.cc:307] RPC server started. Bound to: 127.4.246.253:43459
I20240507 08:12:11.316224 6513 rpcz_store.cc:276] 0507 08:12:06.707064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.707309 (+ 245us) service_pool.cc:225] Handling call
0507 08:12:11.259296 (+4551987us) ts_tablet_manager.cc:1120] Deleting tablet 150efae7834740b798af86ae1dcfe445
0507 08:12:11.259311 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.281030 (+ 21719us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.312680 (+ 31650us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.314205 (+ 1525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551783}
I20240507 08:12:11.318269 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 825) took 4602 ms. Trace:
I20240507 08:12:11.318621 5843 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.318789 5843 ts_tablet_manager.cc:1367] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.005s
I20240507 08:12:11.318938 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 150efae7834740b798af86ae1dcfe445 (table default.Table_1d73a8ee5a4848628ed2472bdc0b084b [id=d95a0f67e809437da87ad837fbf8f2a3]) successfully deleted
I20240507 08:12:11.319305 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69142a1181436abc438211e681c73d (DEFAULT_TABLE table=default.Table_62410585affa4ee1aab69399909e9ff6 [id=a1448cdc66904f45ab32beeb47864e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.319571 5843 raft_consensus.cc:348] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.319803 5843 raft_consensus.cc:374] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.319890 5843 raft_consensus.cc:729] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.320214 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 437fafdae750487ca4655d6a97c230d1 (DEFAULT_TABLE table=default.Table_b43357777b9341f4bbd950ab972819a3 [id=2c264eb4f2324e928abdf2a8f357a5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.320822 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69142a1181436abc438211e681c73d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.321089 5843 consensus_queue.cc:260] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.321651 5843 ts_tablet_manager.cc:1398] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240507 08:12:11.322484 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 451f566fdf2a40a7a61a2d7ae38fffba (DEFAULT_TABLE table=default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.322685 5843 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.322813 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f566fdf2a40a7a61a2d7ae38fffba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.325919 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437fafdae750487ca4655d6a97c230d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.332437 5841 ts_tablet_manager.cc:1909] T 7b4b84ab3f274185a2b838ebb969ab52 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.335184 5843 tablet_bootstrap.cc:654] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.335340 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 832) took 4945 ms. Trace:
I20240507 08:12:11.335465 5841 rpcz_store.cc:276] 0507 08:12:06.390103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.390167 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:11.216023 (+4825856us) ts_tablet_manager.cc:1120] Deleting tablet 7b4b84ab3f274185a2b838ebb969ab52
0507 08:12:11.216031 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.269808 (+ 53777us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.295275 (+ 25467us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.335295 (+ 40020us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825637}
I20240507 08:12:11.335774 5841 tablet_replica.cc:330] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.335963 5841 raft_consensus.cc:2229] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.336117 5841 raft_consensus.cc:2258] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.336153 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7b4b84ab3f274185a2b838ebb969ab52 (table default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]) successfully deleted
I20240507 08:12:11.319603 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 437fafdae750487ca4655d6a97c230d1 (DEFAULT_TABLE table=default.Table_b43357777b9341f4bbd950ab972819a3 [id=2c264eb4f2324e928abdf2a8f357a5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.320438 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 451f566fdf2a40a7a61a2d7ae38fffba (DEFAULT_TABLE table=default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.320921 5840 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.321432 5690 tablet_service.cc:1467] Processing CreateTablet for tablet ce73981ca280473eaa2941447313d7a7 (DEFAULT_TABLE table=default.Table_50b030da7ecf4a98888919bc423e74a0 [id=9b2060aa46304d83a23d248fe0f949bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.336881 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f566fdf2a40a7a61a2d7ae38fffba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.337796 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce73981ca280473eaa2941447313d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.340994 5841 ts_tablet_manager.cc:1875] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.350204 5840 ts_tablet_manager.cc:1367] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.147s user 0.026s sys 0.002s
I20240507 08:12:11.350919 5840 raft_consensus.cc:348] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.351240 5840 raft_consensus.cc:374] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.351337 5840 raft_consensus.cc:729] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.351585 5840 consensus_queue.cc:260] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.352522 5424 tablet_service.cc:1467] Processing CreateTablet for tablet ce73981ca280473eaa2941447313d7a7 (DEFAULT_TABLE table=default.Table_50b030da7ecf4a98888919bc423e74a0 [id=9b2060aa46304d83a23d248fe0f949bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.352941 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce73981ca280473eaa2941447313d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.355894 5840 ts_tablet_manager.cc:1398] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:11.356348 5840 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.356091 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437fafdae750487ca4655d6a97c230d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.357609 6513 rpcz_store.cc:276] 0507 08:12:06.716148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.716205 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:11.318144 (+4601939us) ts_tablet_manager.cc:1120] Deleting tablet 242ea35c61404a91b1211c5ad9207728
0507 08:12:11.318153 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.318253 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4601764}
I20240507 08:12:11.358043 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 827) took 4486 ms. Trace:
I20240507 08:12:11.358134 6513 rpcz_store.cc:276] 0507 08:12:06.871307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.871364 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:11.357926 (+4486562us) ts_tablet_manager.cc:1120] Deleting tablet 96d7d8bcb635416693395b1f9ad27e2f
0507 08:12:11.357934 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.358027 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4486374}
W20240507 08:12:11.358350 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 242ea35c61404a91b1211c5ad9207728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242ea35c61404a91b1211c5ad9207728 already in progress: creating tablet
I20240507 08:12:11.358449 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 829) took 4330 ms. Trace:
W20240507 08:12:11.358775 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 96d7d8bcb635416693395b1f9ad27e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96d7d8bcb635416693395b1f9ad27e2f already in progress: creating tablet
I20240507 08:12:11.358520 6513 rpcz_store.cc:276] 0507 08:12:07.028006 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.028198 (+ 192us) service_pool.cc:225] Handling call
0507 08:12:11.358355 (+4330157us) ts_tablet_manager.cc:1120] Deleting tablet abd47fb41bd14411bb44cc0f25e8cac4
0507 08:12:11.358363 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.358437 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4329962}
I20240507 08:12:11.360051 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 836) took 4300 ms. Trace:
I20240507 08:12:11.360141 6513 rpcz_store.cc:276] 0507 08:12:07.059600 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.060489 (+ 889us) service_pool.cc:225] Handling call
0507 08:12:11.359927 (+4299438us) ts_tablet_manager.cc:1120] Deleting tablet aa485207d0f140c99a9386cb4002c480
0507 08:12:11.359938 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.360034 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4299269}
W20240507 08:12:11.360203 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet abd47fb41bd14411bb44cc0f25e8cac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd47fb41bd14411bb44cc0f25e8cac4 already in progress: creating tablet
I20240507 08:12:11.360487 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 848) took 4146 ms. Trace:
W20240507 08:12:11.360592 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet aa485207d0f140c99a9386cb4002c480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa485207d0f140c99a9386cb4002c480 already in progress: creating tablet
I20240507 08:12:11.360571 6513 rpcz_store.cc:276] 0507 08:12:07.214355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.214846 (+ 491us) service_pool.cc:225] Handling call
0507 08:12:11.360362 (+4145516us) ts_tablet_manager.cc:1120] Deleting tablet b60ec15a83934999871425ef9a64ec72
0507 08:12:11.360373 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.360471 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4139898}
I20240507 08:12:11.360849 6513 tablet_replica.cc:330] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.361073 6513 raft_consensus.cc:2229] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.361243 6513 raft_consensus.cc:2258] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:11.361492 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b60ec15a83934999871425ef9a64ec72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b60ec15a83934999871425ef9a64ec72 already in progress: creating tablet
I20240507 08:12:11.361935 6513 ts_tablet_manager.cc:1875] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.362362 5841 ts_tablet_manager.cc:1888] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:11.365963 5841 log.cc:1196] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/753019050fcf4dc3b69722c3eb475037
I20240507 08:12:11.365891 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4979cf68a0481388253e970bf0161c (DEFAULT_TABLE table=default.Table_9766bca2ee764e50a876dff2ed050158 [id=133d413ace244d39be07293b0c8273b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.366313 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4979cf68a0481388253e970bf0161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.367170 5841 ts_tablet_manager.cc:1909] T 753019050fcf4dc3b69722c3eb475037 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.368032 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 833) took 4953 ms. Trace:
I20240507 08:12:11.368187 5841 rpcz_store.cc:276] 0507 08:12:06.414935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.414994 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:11.335714 (+4920720us) ts_tablet_manager.cc:1120] Deleting tablet 753019050fcf4dc3b69722c3eb475037
0507 08:12:11.335722 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.353324 (+ 17602us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.362215 (+ 8891us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.367992 (+ 5777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4920481}
I20240507 08:12:11.368641 5841 tablet_replica.cc:330] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.368759 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 753019050fcf4dc3b69722c3eb475037 (table default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]) successfully deleted
I20240507 08:12:11.368999 5841 raft_consensus.cc:2229] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.369060 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a6c50d46db8949ee978b87f83a640c0a
I20240507 08:12:11.369761 5841 raft_consensus.cc:2258] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.369825 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a903c17e01c241bcb13787ef551395d2
I20240507 08:12:11.370671 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e561c811a1de48f9bc91175a3d22805a
I20240507 08:12:11.371631 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet d4fac82e12d0437e950d01dbc4117efa from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.372550 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet e561c811a1de48f9bc91175a3d22805a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.372934 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c50d46db8949ee978b87f83a640c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.373620 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/master-1/data/info.pb
I20240507 08:12:11.377951 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet a903c17e01c241bcb13787ef551395d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.379079 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c50d46db8949ee978b87f83a640c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.379400 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet a903c17e01c241bcb13787ef551395d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.382655 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet e561c811a1de48f9bc91175a3d22805a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.383253 5840 tablet_bootstrap.cc:654] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.383374 5843 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.383556 5843 ts_tablet_manager.cc:1367] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.061s user 0.003s sys 0.004s
I20240507 08:12:11.384503 5843 raft_consensus.cc:348] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.384654 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:11.384738 5843 raft_consensus.cc:374] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.384810 5843 raft_consensus.cc:729] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.384881 5083 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskrbFV97/build/asan/bin/kudu as pid 6604
I20240507 08:12:11.385052 5843 consensus_queue.cc:260] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.385100 5261 catalog_manager.cc:5484] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:11.385643 5843 ts_tablet_manager.cc:1398] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240507 08:12:11.385689 5261 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]). Expected version 2 got 0
I20240507 08:12:11.389946 6513 ts_tablet_manager.cc:1888] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:11.390100 6513 log.cc:1196] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e555a89d340448bc852f2441ca850213
I20240507 08:12:11.392580 6513 ts_tablet_manager.cc:1909] T e555a89d340448bc852f2441ca850213 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.393079 5843 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.393318 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 850) took 4167 ms. Trace:
I20240507 08:12:11.393405 6513 rpcz_store.cc:276] 0507 08:12:07.225983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.226042 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:11.360798 (+4134756us) ts_tablet_manager.cc:1120] Deleting tablet e555a89d340448bc852f2441ca850213
0507 08:12:11.360808 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.365156 (+ 4348us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.389830 (+ 24674us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.393300 (+ 3470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4134513}
I20240507 08:12:11.393745 6513 tablet_replica.cc:330] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.393955 6513 raft_consensus.cc:2229] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.394277 6513 raft_consensus.cc:2258] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.394773 5843 tablet_bootstrap.cc:654] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.394825 6513 ts_tablet_manager.cc:1875] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.394738 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e555a89d340448bc852f2441ca850213 (table default.Table_948f06376d4849c79cc3f222a3fcaaad [id=a19f35cb006a4367961f26227bc0a662]) successfully deleted
I20240507 08:12:11.400151 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.400561 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 150efae7834740b798af86ae1dcfe445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.400417 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.404022 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:11.410789 5656 consensus_peers.cc:485] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 753019050fcf4dc3b69722c3eb475037. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.411326 5841 ts_tablet_manager.cc:1875] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.412659 6513 ts_tablet_manager.cc:1888] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.415266 6675 consensus_queue.cc:1035] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:11.435679 6513 log.cc:1196] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/302fbf120e9d472ba6bdf9e6cbdf6dbd
I20240507 08:12:11.437980 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4979cf68a0481388253e970bf0161c (DEFAULT_TABLE table=default.Table_9766bca2ee764e50a876dff2ed050158 [id=133d413ace244d39be07293b0c8273b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.438160 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:11.438526 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4979cf68a0481388253e970bf0161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.445142 5840 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.449298 5840 ts_tablet_manager.cc:1367] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.093s user 0.002s sys 0.011s
I20240507 08:12:11.449548 6513 ts_tablet_manager.cc:1909] T 302fbf120e9d472ba6bdf9e6cbdf6dbd P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.450208 5840 raft_consensus.cc:348] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.451321 5840 raft_consensus.cc:374] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.451393 5840 raft_consensus.cc:729] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.451640 5840 consensus_queue.cc:260] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.452154 5840 ts_tablet_manager.cc:1398] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240507 08:12:11.452564 5840 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.453197 5687 tablet_service.cc:1467] Processing CreateTablet for tablet ccf0944d955e4e4d8b0a11841d72d58e (DEFAULT_TABLE table=default.Table_1b944c710a114e68a10b78be0d257898 [id=873358ed90f74b7b85823bb758e56cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.453616 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf0944d955e4e4d8b0a11841d72d58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.454469 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 096624b740fb4ac88a701a630c97b3eb (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.454831 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096624b740fb4ac88a701a630c97b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.457043 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:11.457727 5841 ts_tablet_manager.cc:1888] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.457875 5841 log.cc:1196] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1c7c39ec9fec4331afabb39d1483ef57
I20240507 08:12:11.460892 5840 tablet_bootstrap.cc:654] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.462074 5421 tablet_service.cc:1467] Processing CreateTablet for tablet ccf0944d955e4e4d8b0a11841d72d58e (DEFAULT_TABLE table=default.Table_1b944c710a114e68a10b78be0d257898 [id=873358ed90f74b7b85823bb758e56cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.463538 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf0944d955e4e4d8b0a11841d72d58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.453181 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 849) took 4227 ms. Trace:
I20240507 08:12:11.465809 6513 rpcz_store.cc:276] 0507 08:12:07.225862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.236294 (+ 10432us) service_pool.cc:225] Handling call
0507 08:12:11.393696 (+4157402us) ts_tablet_manager.cc:1120] Deleting tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd
0507 08:12:11.393706 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.409530 (+ 15824us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.412565 (+ 3035us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.453163 (+ 40598us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4157175}
I20240507 08:12:11.466641 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd (table default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]) successfully deleted
I20240507 08:12:11.467059 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 852) took 4051 ms. Trace:
I20240507 08:12:11.467420 6513 rpcz_store.cc:276] 0507 08:12:07.415248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.415721 (+ 473us) service_pool.cc:225] Handling call
0507 08:12:11.466704 (+4050983us) ts_tablet_manager.cc:1120] Deleting tablet 574a47894d274360891253534bf215a5
0507 08:12:11.466717 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.467040 (+ 323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4050842}
I20240507 08:12:11.468286 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 853) took 4024 ms. Trace:
I20240507 08:12:11.471839 5841 ts_tablet_manager.cc:1909] T 1c7c39ec9fec4331afabb39d1483ef57 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.472995 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 834) took 5053 ms. Trace:
I20240507 08:12:11.473160 5841 rpcz_store.cc:276] 0507 08:12:06.419032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.419090 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:11.368567 (+4949477us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:12:11.368577 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.448681 (+ 80104us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.457624 (+ 8943us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.472940 (+ 15316us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4949268}
I20240507 08:12:11.474666 5841 tablet_replica.cc:330] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.474776 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1c7c39ec9fec4331afabb39d1483ef57 (table default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]) successfully deleted
I20240507 08:12:11.474952 5841 raft_consensus.cc:2229] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:11.476969 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1a4e05273b4f8c83dc8c7825a39310 consensus state has no leader
I20240507 08:12:11.478021 5841 raft_consensus.cc:2258] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.478022 6880 raft_consensus.cc:482] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.478469 6880 raft_consensus.cc:504] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.479142 6880 leader_election.cc:288] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:11.479378 5841 ts_tablet_manager.cc:1875] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.481218 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b6b9232426b948ae8634f7afd65df393
I20240507 08:12:11.481684 5262 catalog_manager.cc:6554] ReplaceTablet: tablet b6b9232426b948ae8634f7afd65df393 deleted and replaced by tablet a1f9ee9f367e42308b3374509bdd6c9f
I20240507 08:12:11.469193 5840 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.482095 7100 consensus_queue.cc:1035] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:11.482267 5840 ts_tablet_manager.cc:1367] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20240507 08:12:11.480083 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.483011 5840 raft_consensus.cc:348] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.483192 5840 raft_consensus.cc:374] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.483249 5840 raft_consensus.cc:729] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:11.483232 5377 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.483446 5840 consensus_queue.cc:260] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.483925 5840 ts_tablet_manager.cc:1398] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:11.484158 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b9232426b948ae8634f7afd65df393 with delete_type TABLET_DATA_DELETED (replaced by tablet a1f9ee9f367e42308b3374509bdd6c9f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.484237 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b9232426b948ae8634f7afd65df393 with delete_type TABLET_DATA_DELETED (replaced by tablet a1f9ee9f367e42308b3374509bdd6c9f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.481693 6513 rpcz_store.cc:276] 0507 08:12:07.444111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.465549 (+ 21438us) service_pool.cc:225] Handling call
0507 08:12:11.468178 (+4002629us) ts_tablet_manager.cc:1120] Deleting tablet dfd9624f4337437bb11b247048e28c08
0507 08:12:11.468189 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.468270 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4000128}
W20240507 08:12:11.484562 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet dfd9624f4337437bb11b247048e28c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd9624f4337437bb11b247048e28c08 already in progress: creating tablet
I20240507 08:12:11.485143 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 851) took 4070 ms. Trace:
I20240507 08:12:11.485234 5840 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.486236 6513 rpcz_store.cc:276] 0507 08:12:07.415136 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.468201 (+ 53065us) service_pool.cc:225] Handling call
0507 08:12:11.485022 (+4016821us) ts_tablet_manager.cc:1120] Deleting tablet 5f0b399a2d074f4ca3626ff17bd5518e
0507 08:12:11.485034 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.485126 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4015725}
W20240507 08:12:11.486929 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5f0b399a2d074f4ca3626ff17bd5518e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0b399a2d074f4ca3626ff17bd5518e already in progress: creating tablet
I20240507 08:12:11.487416 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 855) took 3964 ms. Trace:
I20240507 08:12:11.488222 7201 sys_catalog.cc:263] Verifying existing consensus state
I20240507 08:12:11.487787 6513 rpcz_store.cc:276] 0507 08:12:07.523025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.523075 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:11.487316 (+3964241us) ts_tablet_manager.cc:1120] Deleting tablet 5aa521dd21634cd0af25652931069c41
0507 08:12:11.487325 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.487401 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3963974}
I20240507 08:12:11.490830 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 868) took 3796 ms. Trace:
I20240507 08:12:11.491263 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 482e4d0e762b488b8a0cc4e81c2c821d
I20240507 08:12:11.491325 6513 rpcz_store.cc:276] 0507 08:12:07.694175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.696293 (+ 2118us) service_pool.cc:225] Handling call
0507 08:12:11.490731 (+3794438us) ts_tablet_manager.cc:1120] Deleting tablet bc029bc110b847f79d876ec3ab568832
0507 08:12:11.490740 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.490815 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3794124}
I20240507 08:12:11.491652 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 867) took 3797 ms. Trace:
I20240507 08:12:11.491716 6513 rpcz_store.cc:276] 0507 08:12:07.693957 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.696685 (+ 2728us) service_pool.cc:225] Handling call
0507 08:12:11.491553 (+3794868us) ts_tablet_manager.cc:1120] Deleting tablet 32ed504c4cf443e48ab285466b9e9b5d
0507 08:12:11.491562 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.491639 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3794241}
I20240507 08:12:11.491976 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 866) took 3798 ms. Trace:
I20240507 08:12:11.492031 6513 rpcz_store.cc:276] 0507 08:12:07.693768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.698551 (+ 4783us) service_pool.cc:225] Handling call
0507 08:12:11.491892 (+3793341us) ts_tablet_manager.cc:1120] Deleting tablet fccc127d380d4035905c1b152ab1017b
0507 08:12:11.491900 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.491963 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3793022}
W20240507 08:12:11.493553 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5aa521dd21634cd0af25652931069c41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5aa521dd21634cd0af25652931069c41 already in progress: creating tablet
I20240507 08:12:11.493636 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.297245 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.299036 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 4e69142a1181436abc438211e681c73d (DEFAULT_TABLE table=default.Table_62410585affa4ee1aab69399909e9ff6 [id=a1448cdc66904f45ab32beeb47864e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.303768 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28e5a380f2784e3a806afcbb5cd2519a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.494442 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e69142a1181436abc438211e681c73d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:11.496944 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bc029bc110b847f79d876ec3ab568832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc029bc110b847f79d876ec3ab568832 already in progress: creating tablet
I20240507 08:12:11.496965 6675 raft_consensus.cc:482] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:11.497241 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 32ed504c4cf443e48ab285466b9e9b5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32ed504c4cf443e48ab285466b9e9b5d already in progress: creating tablet
I20240507 08:12:11.497267 6675 raft_consensus.cc:504] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.321857 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 451f566fdf2a40a7a61a2d7ae38fffba (DEFAULT_TABLE table=default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.497833 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 451f566fdf2a40a7a61a2d7ae38fffba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.497845 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 096624b740fb4ac88a701a630c97b3eb (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.498230 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096624b740fb4ac88a701a630c97b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.498478 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.498695 5447 raft_consensus.cc:2454] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.499050 5656 leader_election.cc:302] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.499274 7095 raft_consensus.cc:2781] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.499328 7095 raft_consensus.cc:482] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.499380 7095 raft_consensus.cc:3037] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.500300 5841 ts_tablet_manager.cc:1888] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.500447 5841 log.cc:1196] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:12:11.500803 5841 ts_tablet_manager.cc:1909] T 03a59c7257e74b4abaa4ea7a1ee9136d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.501099 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 835) took 5080 ms. Trace:
I20240507 08:12:11.501767 7095 raft_consensus.cc:504] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.321866 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 437fafdae750487ca4655d6a97c230d1 (DEFAULT_TABLE table=default.Table_b43357777b9341f4bbd950ab972819a3 [id=2c264eb4f2324e928abdf2a8f357a5f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.502311 7095 leader_election.cc:288] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.502367 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 437fafdae750487ca4655d6a97c230d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.503082 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.503244 5447 raft_consensus.cc:3037] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.505208 7201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap starting.
I20240507 08:12:11.322178 5555 tablet_service.cc:1467] Processing CreateTablet for tablet ce73981ca280473eaa2941447313d7a7 (DEFAULT_TABLE table=default.Table_50b030da7ecf4a98888919bc423e74a0 [id=9b2060aa46304d83a23d248fe0f949bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.497848 6675 leader_election.cc:288] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.505841 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce73981ca280473eaa2941447313d7a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.487903 5840 tablet_bootstrap.cc:654] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.507185 5447 raft_consensus.cc:2454] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.507593 5656 leader_election.cc:302] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.507898 7095 raft_consensus.cc:2781] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.508237 7095 raft_consensus.cc:686] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.508509 7095 consensus_queue.cc:237] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.344218 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 8c4979cf68a0481388253e970bf0161c (DEFAULT_TABLE table=default.Table_9766bca2ee764e50a876dff2ed050158 [id=133d413ace244d39be07293b0c8273b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.366820 5839 tablet_bootstrap.cc:492] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.378921 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet a6c50d46db8949ee978b87f83a640c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.379084 7044 raft_consensus.cc:482] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.509743 5839 ts_tablet_manager.cc:1367] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.254s user 0.003s sys 0.010s
I20240507 08:12:11.509881 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c4979cf68a0481388253e970bf0161c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.509820 7044 raft_consensus.cc:504] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.510519 5839 raft_consensus.cc:348] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.379251 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet a903c17e01c241bcb13787ef551395d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.510722 5839 raft_consensus.cc:374] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.510856 5839 raft_consensus.cc:729] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.511080 5839 consensus_queue.cc:260] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.385345 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet e561c811a1de48f9bc91175a3d22805a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.437980 5547 tablet_service.cc:1467] Processing CreateTablet for tablet ccf0944d955e4e4d8b0a11841d72d58e (DEFAULT_TABLE table=default.Table_1b944c710a114e68a10b78be0d257898 [id=873358ed90f74b7b85823bb758e56cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.511716 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccf0944d955e4e4d8b0a11841d72d58e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.411692 5579 raft_consensus.cc:3037] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.514299 6675 raft_consensus.cc:482] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.514425 6675 raft_consensus.cc:504] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.514613 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.472252 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 096624b740fb4ac88a701a630c97b3eb (DEFAULT_TABLE table=default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2 [id=d25553f0f183471cbbb32a1ad2876fdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:11.515317 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 096624b740fb4ac88a701a630c97b3eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:11.515673 6675 leader_election.cc:288] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.448211 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:11.483978 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet b6b9232426b948ae8634f7afd65df393 with delete_type TABLET_DATA_DELETED (replaced by tablet a1f9ee9f367e42308b3374509bdd6c9f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.517117 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.493701 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.510584 7044 leader_election.cc:288] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.480275 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.518157 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:11.518271 5378 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.518374 5378 leader_election.cc:302] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.518584 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "835d7b779d724df4863bb13a5140dbf0" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.519157 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.519383 5447 raft_consensus.cc:2454] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.522419 5517 leader_election.cc:302] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:11.522694 7044 raft_consensus.cc:2726] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.523326 7107 raft_consensus.cc:2726] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.523468 5656 leader_election.cc:302] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:11.524421 5378 consensus_peers.cc:485] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.531356 7100 raft_consensus.cc:2781] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.531486 7100 raft_consensus.cc:482] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.531565 7100 raft_consensus.cc:3037] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.531751 5839 ts_tablet_manager.cc:1398] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.022s user 0.001s sys 0.008s
I20240507 08:12:11.532083 5839 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.533063 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 482e4d0e762b488b8a0cc4e81c2c821d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.533532 7055 tablet.cc:1722] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:11.534189 7100 raft_consensus.cc:504] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.535020 7100 leader_election.cc:288] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.533883 5839 tablet_bootstrap.cc:654] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.539543 6880 raft_consensus.cc:482] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.539691 6880 raft_consensus.cc:504] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.540045 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.540251 5447 raft_consensus.cc:3037] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.540351 6880 leader_election.cc:288] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:11.537386 6784 raft_consensus.cc:482] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.540690 6784 raft_consensus.cc:504] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.541342 6784 leader_election.cc:288] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.541826 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.542042 5437 raft_consensus.cc:2454] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.543231 5656 leader_election.cc:302] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:11.543561 6912 raft_consensus.cc:2781] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.543632 6912 raft_consensus.cc:482] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.543689 6912 raft_consensus.cc:3037] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.544539 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.545114 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet fccc127d380d4035905c1b152ab1017b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fccc127d380d4035905c1b152ab1017b already in progress: creating tablet
W20240507 08:12:11.545545 5378 leader_election.cc:341] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.545817 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.538851 5654 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.546533 5654 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.546586 6912 raft_consensus.cc:504] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:11.546793 5654 leader_election.cc:341] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.547048 5654 leader_election.cc:341] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.547212 6912 leader_election.cc:288] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:11.547472 5654 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.548379 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:11.548807 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31251576beee4a8a81a453b721c9c058" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.548815 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:11.549198 5654 leader_election.cc:341] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.549528 5654 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.549844 5654 leader_election.cc:341] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.550110 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.550280 5437 raft_consensus.cc:3037] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.553085 5437 raft_consensus.cc:2454] T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.555274 5656 leader_election.cc:302] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.555948 6784 raft_consensus.cc:2781] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.556088 6784 raft_consensus.cc:686] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.556370 6784 consensus_queue.cc:237] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.558795 7079 log.cc:1196] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4e63c3f3ecc84913ad05864be450e9b0
I20240507 08:12:11.561668 5447 raft_consensus.cc:2454] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.563412 5656 leader_election.cc:302] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.563673 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c1bf63edee4348b1aad5f47c7fe5b726"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:11.563880 6675 raft_consensus.cc:2781] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.563987 6675 raft_consensus.cc:686] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:11.564244 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1bf63edee4348b1aad5f47c7fe5b726 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.564195 6675 consensus_queue.cc:237] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.565505 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_99f001a5e47f44928eb0d9d255bd7423"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.565974 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99f001a5e47f44928eb0d9d255bd7423 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.567206 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2468219608d6479fba6a3ef7216bd1a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.567605 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2468219608d6479fba6a3ef7216bd1a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.568737 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6bd4fbf9fea6459893b5a20cf44ff554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.569144 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bd4fbf9fea6459893b5a20cf44ff554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.572384 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]). Expected version 2 got 0
I20240507 08:12:11.577910 6912 consensus_queue.cc:1035] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:11.583530 5839 tablet_bootstrap.cc:492] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.583781 5839 ts_tablet_manager.cc:1367] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.001s
W20240507 08:12:11.584614 5654 consensus_peers.cc:485] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.584601 5839 raft_consensus.cc:348] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.584970 5839 raft_consensus.cc:374] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.585058 5839 raft_consensus.cc:729] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.585253 5840 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.585300 5839 consensus_queue.cc:260] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.585453 5840 ts_tablet_manager.cc:1367] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.005s
I20240507 08:12:11.588058 5839 ts_tablet_manager.cc:1398] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:11.588376 5839 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.588501 7079 ts_tablet_manager.cc:1909] T 4e63c3f3ecc84913ad05864be450e9b0 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:11.589349 7079 tablet_replica.cc:330] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.589555 7079 raft_consensus.cc:2229] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.573422 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8cf67af06cf44e458ec7c70c6a18ffd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.593828 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cf67af06cf44e458ec7c70c6a18ffd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.593812 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.589138 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4e63c3f3ecc84913ad05864be450e9b0 (table default.Table_3da110b2f7014fdca83ebeeb4c70c3d1 [id=352055be23d04485b3896bd169ea07ba]) successfully deleted
I20240507 08:12:11.573809 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_57ae6d7b295a4fb38adfb6579b01dc83" } new_table_name: "default.Table_ca857f7ec80945ff80de395cbcd59f8d" modify_external_catalogs: true
I20240507 08:12:11.594429 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 886) took 3232 ms. Trace:
W20240507 08:12:11.594532 5377 leader_election.cc:341] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.594647 5377 leader_election.cc:302] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.594554 6513 rpcz_store.cc:276] 0507 08:12:08.361969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.480093 (+118124us) service_pool.cc:225] Handling call
0507 08:12:11.521038 (+3040945us) ts_tablet_manager.cc:1120] Deleting tablet 8005100f64fd49a3ac29d34aa728fdfb
0507 08:12:11.521049 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.594411 (+ 73362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3040693}
I20240507 08:12:11.595149 7107 raft_consensus.cc:2726] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:11.596356 5654 consensus_peers.cc:485] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.599998 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 887) took 3237 ms. Trace:
I20240507 08:12:11.600093 6513 rpcz_store.cc:276] 0507 08:12:08.362048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.480356 (+118308us) service_pool.cc:225] Handling call
0507 08:12:11.594801 (+3114445us) ts_tablet_manager.cc:1120] Deleting tablet a570e39a52d84d3dbcaeb9ea005d9f41
0507 08:12:11.599876 (+ 5075us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.599982 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3114314}
I20240507 08:12:11.600452 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 888) took 3238 ms. Trace:
W20240507 08:12:11.600558 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a570e39a52d84d3dbcaeb9ea005d9f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a570e39a52d84d3dbcaeb9ea005d9f41 already in progress: creating tablet
I20240507 08:12:11.600535 6513 rpcz_store.cc:276] 0507 08:12:08.362138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.480496 (+118358us) service_pool.cc:225] Handling call
0507 08:12:11.600352 (+3119856us) ts_tablet_manager.cc:1120] Deleting tablet c40ed1afee294143ab85a8c03d080b74
0507 08:12:11.600360 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.600436 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3119738}
I20240507 08:12:11.600843 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 889) took 3238 ms. Trace:
I20240507 08:12:11.600912 6513 rpcz_store.cc:276] 0507 08:12:08.362210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.480623 (+118413us) service_pool.cc:225] Handling call
0507 08:12:11.600743 (+3120120us) ts_tablet_manager.cc:1120] Deleting tablet c7bb96bb9d8f4930b531a3dd30d733c8
0507 08:12:11.600752 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.600827 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3120010}
I20240507 08:12:11.601231 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 911) took 2979 ms. Trace:
I20240507 08:12:11.601313 6513 rpcz_store.cc:276] 0507 08:12:08.621883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955027 (+333144us) service_pool.cc:225] Handling call
0507 08:12:11.601128 (+2646101us) ts_tablet_manager.cc:1120] Deleting tablet 182124e13ca749c8a2a96d38514158b1
0507 08:12:11.601138 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.601216 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2645943}
I20240507 08:12:11.601596 6513 tablet_replica.cc:330] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.601930 6513 raft_consensus.cc:2229] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.602310 6513 raft_consensus.cc:2258] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:11.602502 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c40ed1afee294143ab85a8c03d080b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40ed1afee294143ab85a8c03d080b74 already in progress: creating tablet
W20240507 08:12:11.603025 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c7bb96bb9d8f4930b531a3dd30d733c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7bb96bb9d8f4930b531a3dd30d733c8 already in progress: creating tablet
W20240507 08:12:11.603305 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 182124e13ca749c8a2a96d38514158b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182124e13ca749c8a2a96d38514158b1 already in progress: creating tablet
I20240507 08:12:11.603025 5840 raft_consensus.cc:348] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.603679 5840 raft_consensus.cc:374] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.603785 5840 raft_consensus.cc:729] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.595396 5839 tablet_bootstrap.cc:654] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.604976 5840 consensus_queue.cc:260] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.612072 7079 raft_consensus.cc:2258] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:11.619915 5654 consensus_peers.cc:485] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.629858 5840 ts_tablet_manager.cc:1398] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.044s user 0.002s sys 0.000s
I20240507 08:12:11.630209 5840 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.641498 7079 ts_tablet_manager.cc:1875] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:11.645221 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 5aca2077b76b49e3bf9daae57dc09b8f (table default.Table_9383fc310d0f4a6fb20a0c555d0e8257 [id=075087d8f9af4e189de5ff3e7004e78b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5aca2077b76b49e3bf9daae57dc09b8f consensus state has no leader
I20240507 08:12:11.650210 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.650362 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.650465 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.650916 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.651945 6880 raft_consensus.cc:482] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.652076 6880 raft_consensus.cc:504] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:11.652688 6880 leader_election.cc:288] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:11.652981 5656 consensus_peers.cc:485] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7b4b84ab3f274185a2b838ebb969ab52. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.653086 5843 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.653199 5843 ts_tablet_manager.cc:1367] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.260s user 0.021s sys 0.008s
I20240507 08:12:11.653354 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.653767 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.653837 5377 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.654367 5378 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.654457 5378 leader_election.cc:302] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.654404 5843 raft_consensus.cc:348] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.654572 5843 raft_consensus.cc:374] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.654618 5843 raft_consensus.cc:729] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.654868 6880 raft_consensus.cc:2726] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.654821 5843 consensus_queue.cc:260] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.655021 6513 ts_tablet_manager.cc:1875] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.655479 5843 ts_tablet_manager.cc:1398] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:11.655722 5843 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.657140 5843 tablet_bootstrap.cc:654] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.667172 5839 tablet_bootstrap.cc:492] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.667312 5839 ts_tablet_manager.cc:1367] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.003s
I20240507 08:12:11.667996 5839 raft_consensus.cc:348] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.668186 5839 raft_consensus.cc:374] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.668236 5839 raft_consensus.cc:729] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.668481 5839 consensus_queue.cc:260] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.670258 5840 tablet_bootstrap.cc:654] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.671602 5839 ts_tablet_manager.cc:1398] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:11.671870 5839 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.673142 5839 tablet_bootstrap.cc:654] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.684059 5584 raft_consensus.cc:3037] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.692373 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 46cc8597ca5042ca8e85d99803d07206
I20240507 08:12:11.695988 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:11.698379 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cc8597ca5042ca8e85d99803d07206 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.698388 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.700181 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d4fac82e12d0437e950d01dbc4117efa
I20240507 08:12:11.700233 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cc8597ca5042ca8e85d99803d07206 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.700649 5257 catalog_manager.cc:6554] ReplaceTablet: tablet d4fac82e12d0437e950d01dbc4117efa deleted and replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95
I20240507 08:12:11.701025 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d0672e8f460d4a779e4defb93cbbff2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.701606 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0672e8f460d4a779e4defb93cbbff2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.701534 6513 ts_tablet_manager.cc:1888] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.702179 6513 log.cc:1196] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:12:11.702448 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fac82e12d0437e950d01dbc4117efa with delete_type TABLET_DATA_DELETED (replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.703044 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fac82e12d0437e950d01dbc4117efa with delete_type TABLET_DATA_DELETED (replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.703214 6513 ts_tablet_manager.cc:1909] T 3b5e3d23bbb6439a87b303dff1b44a1d P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.704106 5261 catalog_manager.cc:5484] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:11.704669 5261 catalog_manager.cc:5484] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:11.705041 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
I20240507 08:12:11.705636 7079 ts_tablet_manager.cc:1888] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:11.705744 7079 log.cc:1196] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/98aa4ec7b93943e5bf8f9758b3bb71e0
I20240507 08:12:11.706082 7079 ts_tablet_manager.cc:1909] T 98aa4ec7b93943e5bf8f9758b3bb71e0 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:11.706774 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 98aa4ec7b93943e5bf8f9758b3bb71e0 (table default.Table_41b539df8107441aa4610f0fc94729c2 [id=277f3a853f9f40b18fd368cb8f2e9918]) successfully deleted
I20240507 08:12:11.707190 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 46cc8597ca5042ca8e85d99803d07206 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:11.707418 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a6c50d46db8949ee978b87f83a640c0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c50d46db8949ee978b87f83a640c0a already in progress: creating tablet
W20240507 08:12:11.708002 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a903c17e01c241bcb13787ef551395d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a903c17e01c241bcb13787ef551395d2 already in progress: creating tablet
W20240507 08:12:11.708545 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e561c811a1de48f9bc91175a3d22805a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e561c811a1de48f9bc91175a3d22805a already in progress: creating tablet
W20240507 08:12:11.709107 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b6b9232426b948ae8634f7afd65df393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6b9232426b948ae8634f7afd65df393 already in progress: creating tablet
W20240507 08:12:11.709674 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 482e4d0e762b488b8a0cc4e81c2c821d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 482e4d0e762b488b8a0cc4e81c2c821d already in progress: creating tablet
W20240507 08:12:11.710229 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 46cc8597ca5042ca8e85d99803d07206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cc8597ca5042ca8e85d99803d07206 already in progress: creating tablet
I20240507 08:12:11.710932 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5eb952438aa148978bdecbc42cb74318"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:11.710974 6880 consensus_queue.cc:1035] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:11.711382 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5eb952438aa148978bdecbc42cb74318 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.712878 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]). Expected version 1 got 0
I20240507 08:12:11.714148 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.717736 5841 rpcz_store.cc:276] 0507 08:12:06.420658 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.420716 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:11.474593 (+5053877us) ts_tablet_manager.cc:1120] Deleting tablet 03a59c7257e74b4abaa4ea7a1ee9136d
0507 08:12:11.474601 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.489621 (+ 15020us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.500189 (+ 10568us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.501066 (+ 877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5053681}
I20240507 08:12:11.718346 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.718587 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 03a59c7257e74b4abaa4ea7a1ee9136d (table default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]) successfully deleted
I20240507 08:12:11.718858 5841 tablet_replica.cc:330] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.719069 5841 raft_consensus.cc:2229] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.719226 5841 raft_consensus.cc:2258] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.720058 5841 ts_tablet_manager.cc:1875] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.725023 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fac82e12d0437e950d01dbc4117efa with delete_type TABLET_DATA_DELETED (replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.732465 7079 tablet_replica.cc:330] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.733748 5839 tablet_bootstrap.cc:492] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.733980 5839 ts_tablet_manager.cc:1367] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.000s
I20240507 08:12:11.734691 5839 raft_consensus.cc:348] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.734944 5839 raft_consensus.cc:374] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.735051 5839 raft_consensus.cc:729] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.735309 5839 consensus_queue.cc:260] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.735893 5839 ts_tablet_manager.cc:1398] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:11.736236 5839 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.739810 5839 tablet_bootstrap.cc:654] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.740537 7079 raft_consensus.cc:2229] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.740957 7079 raft_consensus.cc:2258] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.741180 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 912) took 3082 ms. Trace:
I20240507 08:12:11.741322 6513 rpcz_store.cc:276] 0507 08:12:08.621990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955195 (+333205us) service_pool.cc:225] Handling call
0507 08:12:11.601538 (+2646343us) ts_tablet_manager.cc:1120] Deleting tablet 3b5e3d23bbb6439a87b303dff1b44a1d
0507 08:12:11.601546 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.661289 (+ 59743us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.701433 (+ 40144us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.704121 (+ 2688us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2646242}
I20240507 08:12:11.743029 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 3b5e3d23bbb6439a87b303dff1b44a1d (table default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]) successfully deleted
I20240507 08:12:11.744032 6675 raft_consensus.cc:482] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.744163 6675 raft_consensus.cc:504] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.744767 6675 leader_election.cc:288] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.745379 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.745905 5654 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.746033 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:11.746246 5443 raft_consensus.cc:2454] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:11.747313 5656 leader_election.cc:302] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.747855 6675 raft_consensus.cc:2781] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.747939 6675 raft_consensus.cc:482] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.748006 6675 raft_consensus.cc:3037] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.750687 6675 raft_consensus.cc:504] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.751488 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_56208b251bcf49dc8b79aac94c9ca7b6" } new_table_name: "default.Table_908d49dcd5814349aee9767384232deb" modify_external_catalogs: true
I20240507 08:12:11.751852 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:11.752329 5654 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.753134 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3a4888c1005a4a4783fa36c86a79c2f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.753665 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a4888c1005a4a4783fa36c86a79c2f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.754891 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f1c86fd506164b62a5000e951232875e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.755333 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1c86fd506164b62a5000e951232875e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.755702 7101 consensus_queue.cc:1035] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:11.756449 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d0ecbb17529c4707a7b3cfd97b764d1c" } new_table_name: "default.Table_03e9834c587b47ff9dcd33041dba5abf" modify_external_catalogs: true
I20240507 08:12:11.757879 6675 leader_election.cc:288] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:11.750672 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 913) took 3128 ms. Trace:
I20240507 08:12:11.758148 6513 rpcz_store.cc:276] 0507 08:12:08.622091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955303 (+333212us) service_pool.cc:225] Handling call
0507 08:12:11.750577 (+2795274us) ts_tablet_manager.cc:1120] Deleting tablet 4716c1285e91432a92b25ab13ada2779
0507 08:12:11.750588 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.750656 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2795176}
I20240507 08:12:11.758570 6513 tablet_replica.cc:330] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:11.758781 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4716c1285e91432a92b25ab13ada2779 because the tablet was not found. No further retry: Not found: Tablet not found: 4716c1285e91432a92b25ab13ada2779
I20240507 08:12:11.758811 6513 raft_consensus.cc:2229] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.758826 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a02a5a493bc47cdbb6f7bd5b5c28e7f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:11.758993 5439 raft_consensus.cc:3037] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.759192 6513 raft_consensus.cc:2258] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.760398 6513 ts_tablet_manager.cc:1875] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.762082 5439 raft_consensus.cc:2454] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:11.762468 5656 leader_election.cc:302] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:11.763011 5656 consensus_peers.cc:485] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c7c39ec9fec4331afabb39d1483ef57. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.763170 6675 raft_consensus.cc:2781] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.763301 6675 raft_consensus.cc:686] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:11.763584 6675 consensus_queue.cc:237] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.767381 5584 raft_consensus.cc:3037] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.768996 7079 ts_tablet_manager.cc:1875] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.773851 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:11.776099 7079 ts_tablet_manager.cc:1888] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.784752 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_69d9a0c148964285a59fc90ed39304dd" } new_table_name: "default.Table_0a31f4f88e624f448dfcb3dd6bcc825d" modify_external_catalogs: true
I20240507 08:12:11.786350 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet aafd1dc2eaf84390b072f01a9e76232d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.785251 5843 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.786806 5840 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.788048 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_630ea5aaa63747deaf53c4b3b58cf2be" } modify_external_catalogs: true
I20240507 08:12:11.787882 5840 ts_tablet_manager.cc:1367] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.158s user 0.001s sys 0.015s
I20240507 08:12:11.788175 5843 ts_tablet_manager.cc:1367] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.132s user 0.003s sys 0.013s
I20240507 08:12:11.788049 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_45f5c047857b4e9fbcdc45084bdf09a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.792588 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45f5c047857b4e9fbcdc45084bdf09a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.796696 5841 ts_tablet_manager.cc:1888] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.796859 5841 log.cc:1196] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/a3e58ba8c26d4f17af24eb379aca1f60
I20240507 08:12:11.797240 5841 ts_tablet_manager.cc:1909] T a3e58ba8c26d4f17af24eb379aca1f60 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.798974 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 836) took 5377 ms. Trace:
I20240507 08:12:11.801394 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
W20240507 08:12:11.802557 5654 consensus_peers.cc:485] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:11.807538 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.807528 5840 raft_consensus.cc:348] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.808967 5840 raft_consensus.cc:374] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.809543 5840 raft_consensus.cc:729] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.788254 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_630ea5aaa63747deaf53c4b3b58cf2be" } modify_external_catalogs: true
I20240507 08:12:11.811264 5840 consensus_queue.cc:260] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.814985 6513 ts_tablet_manager.cc:1888] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.815243 6513 log.cc:1196] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:12:11.816100 6513 ts_tablet_manager.cc:1909] T 03a59c7257e74b4abaa4ea7a1ee9136d P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:11.817420 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 914) took 3195 ms. Trace:
I20240507 08:12:11.802980 5843 raft_consensus.cc:348] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.817865 5843 raft_consensus.cc:374] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.817991 5843 raft_consensus.cc:729] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.818264 5843 consensus_queue.cc:260] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.820601 5843 ts_tablet_manager.cc:1398] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20240507 08:12:11.820906 5843 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:11.820896 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 669ffac4b4f34551ba930f0a259e78a6 has no consensus state
I20240507 08:12:11.821331 5840 ts_tablet_manager.cc:1398] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
W20240507 08:12:11.822175 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 consensus state has no leader
I20240507 08:12:11.822841 5843 tablet_bootstrap.cc:654] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.817749 5841 rpcz_store.cc:276] 0507 08:12:06.421388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.421442 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:11.718814 (+5297372us) ts_tablet_manager.cc:1120] Deleting tablet a3e58ba8c26d4f17af24eb379aca1f60
0507 08:12:11.718823 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.723041 (+ 4218us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.796602 (+ 73561us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.798953 (+ 2351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5297174}
I20240507 08:12:11.823602 5841 tablet_replica.cc:330] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.823774 5841 raft_consensus.cc:2229] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.824069 5841 raft_consensus.cc:2258] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.824596 5841 ts_tablet_manager.cc:1875] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.787125 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet a3e58ba8c26d4f17af24eb379aca1f60 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.787706 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b4b84ab3f274185a2b838ebb969ab52 with delete_type TABLET_DATA_DELETED (replaced by tablet fed715ba993a4f0c9e54b209ef031981) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.798187 5839 tablet_bootstrap.cc:492] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.825726 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet a3e58ba8c26d4f17af24eb379aca1f60 (table default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]) successfully deleted
I20240507 08:12:11.825546 5839 ts_tablet_manager.cc:1367] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20240507 08:12:11.826637 5840 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.826876 5839 raft_consensus.cc:348] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.827072 5839 raft_consensus.cc:374] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.827210 5839 raft_consensus.cc:729] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.827548 5839 consensus_queue.cc:260] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.828248 5839 ts_tablet_manager.cc:1398] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:11.828608 5839 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.829380 5840 tablet_bootstrap.cc:654] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.830204 5839 tablet_bootstrap.cc:654] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.828379 7079 log.cc:1196] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d6f8f4205f1f48abaf4fcd7113ad44a8
I20240507 08:12:11.835848 7079 ts_tablet_manager.cc:1909] T d6f8f4205f1f48abaf4fcd7113ad44a8 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:11.838490 7079 tablet_replica.cc:330] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:11.838833 7079 raft_consensus.cc:2229] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.839093 7079 raft_consensus.cc:2258] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.841122 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_953eb3a1426a4a479cb256cc68e6d4be" } modify_external_catalogs: true
I20240507 08:12:11.841325 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_953eb3a1426a4a479cb256cc68e6d4be" } modify_external_catalogs: true
I20240507 08:12:11.842828 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_375f890255ed48659c41d4c11bb30578" } modify_external_catalogs: true
I20240507 08:12:11.843003 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_375f890255ed48659c41d4c11bb30578" } modify_external_catalogs: true
I20240507 08:12:11.844128 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:11.844493 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet cacfe980b2aa4a60ad05bbe102a94266 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.845499 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b3fefd616dbe4d5282b1e4325f179875" } new_table_name: "default.Table_18f782a640e348628a86c4df1dcca11c" modify_external_catalogs: true
I20240507 08:12:11.846563 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (table default.Table_87bb488658cd4819a51ff1ff0993ceab [id=799497aabd9b4f17bc42b655e947b3ed]) successfully deleted
W20240507 08:12:11.847023 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d4fac82e12d0437e950d01dbc4117efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4fac82e12d0437e950d01dbc4117efa already in progress: creating tablet
I20240507 08:12:11.848037 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.851796 5260 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:11.852878 6880 raft_consensus.cc:482] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.853010 6880 raft_consensus.cc:504] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.853636 6880 leader_election.cc:288] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:11.854269 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.854676 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.854915 5377 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:11.855598 5378 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.855708 5378 leader_election.cc:302] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.856132 6880 raft_consensus.cc:2726] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.857728 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 97c978ec8bca4aa5b2ffaf22c3290747 with delete_type TABLET_DATA_DELETED (replaced by tablet aa806508ec3e489a9b2c2fa9196198d5) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.858237 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a69f8756b5934280bf92b31a383e0f88" } modify_external_catalogs: true
I20240507 08:12:11.860137 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:11.860348 5654 consensus_peers.cc:485] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:11.866815 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13fa1a807d3548d6836652379d455b0d consensus state has no leader
I20240507 08:12:11.867131 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 6ad4ed7878454fea930b3982bcb6eed4 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.872241 5584 raft_consensus.cc:3037] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.872399 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aafd1dc2eaf84390b072f01a9e76232d
I20240507 08:12:11.872848 5257 catalog_manager.cc:6554] ReplaceTablet: tablet aafd1dc2eaf84390b072f01a9e76232d deleted and replaced by tablet 82c30b345f9a4979b17af3c3d71f6f54
I20240507 08:12:11.873510 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e3783fc7fe434696898ba14ab77a9058
I20240507 08:12:11.874308 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet e3783fc7fe434696898ba14ab77a9058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.875150 5841 ts_tablet_manager.cc:1888] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.875717 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet e3783fc7fe434696898ba14ab77a9058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.875851 5841 log.cc:1196] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4e63c3f3ecc84913ad05864be450e9b0
I20240507 08:12:11.875998 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet aafd1dc2eaf84390b072f01a9e76232d with delete_type TABLET_DATA_DELETED (replaced by tablet 82c30b345f9a4979b17af3c3d71f6f54) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.876299 5841 ts_tablet_manager.cc:1909] T 4e63c3f3ecc84913ad05864be450e9b0 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.877172 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet e3783fc7fe434696898ba14ab77a9058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.877228 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet aafd1dc2eaf84390b072f01a9e76232d with delete_type TABLET_DATA_DELETED (replaced by tablet 82c30b345f9a4979b17af3c3d71f6f54) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.880398 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.880180 6513 rpcz_store.cc:276] 0507 08:12:08.622177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955403 (+333226us) service_pool.cc:225] Handling call
0507 08:12:11.758520 (+2803117us) ts_tablet_manager.cc:1120] Deleting tablet 03a59c7257e74b4abaa4ea7a1ee9136d
0507 08:12:11.758530 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.791301 (+ 32771us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.814743 (+ 23442us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.817353 (+ 2610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2803004}
I20240507 08:12:11.880779 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 03a59c7257e74b4abaa4ea7a1ee9136d (table default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]) successfully deleted
I20240507 08:12:11.877741 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet aafd1dc2eaf84390b072f01a9e76232d with delete_type TABLET_DATA_DELETED (replaced by tablet 82c30b345f9a4979b17af3c3d71f6f54) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.882954 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 837) took 5460 ms. Trace:
I20240507 08:12:11.883059 5841 rpcz_store.cc:276] 0507 08:12:06.422118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.422172 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:11.823558 (+5401386us) ts_tablet_manager.cc:1120] Deleting tablet 4e63c3f3ecc84913ad05864be450e9b0
0507 08:12:11.823567 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.839279 (+ 15712us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.874990 (+ 35711us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.882937 (+ 7947us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5401191}
I20240507 08:12:11.884470 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4e63c3f3ecc84913ad05864be450e9b0 (table default.Table_3da110b2f7014fdca83ebeeb4c70c3d1 [id=352055be23d04485b3896bd169ea07ba]) successfully deleted
I20240507 08:12:11.884812 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cacfe980b2aa4a60ad05bbe102a94266
I20240507 08:12:11.885234 5259 catalog_manager.cc:6554] ReplaceTablet: tablet cacfe980b2aa4a60ad05bbe102a94266 deleted and replaced by tablet d8e03f8655714d309d5c8ae49c4b314b
I20240507 08:12:11.885939 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 323b96ae7d1e471998a39f3be0857ce7
I20240507 08:12:11.890352 5841 tablet_replica.cc:330] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.890607 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.891187 5841 raft_consensus.cc:2229] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.891927 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.892021 5841 raft_consensus.cc:2258] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.892227 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 with delete_type TABLET_DATA_DELETED (replaced by tablet d8e03f8655714d309d5c8ae49c4b314b) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.892758 5841 ts_tablet_manager.cc:1875] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.894030 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 323b96ae7d1e471998a39f3be0857ce7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.894223 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 with delete_type TABLET_DATA_DELETED (replaced by tablet d8e03f8655714d309d5c8ae49c4b314b) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.895562 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet cacfe980b2aa4a60ad05bbe102a94266 with delete_type TABLET_DATA_DELETED (replaced by tablet d8e03f8655714d309d5c8ae49c4b314b) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.899542 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6ad4ed7878454fea930b3982bcb6eed4
W20240507 08:12:11.899629 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d7e215266f54e27a76f813f517b7ac5 consensus state has no leader
I20240507 08:12:11.900329 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 6ad4ed7878454fea930b3982bcb6eed4 deleted and replaced by tablet a68badb3b5644ed8b007b2b04bc4f6e1
I20240507 08:12:11.900419 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 with delete_type TABLET_DATA_DELETED (replaced by tablet a68badb3b5644ed8b007b2b04bc4f6e1) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.901216 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 with delete_type TABLET_DATA_DELETED (replaced by tablet a68badb3b5644ed8b007b2b04bc4f6e1) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.902616 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_889601b1d99641d8b8420888cf849ec1" } modify_external_catalogs: true
I20240507 08:12:11.903177 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ad4ed7878454fea930b3982bcb6eed4 with delete_type TABLET_DATA_DELETED (replaced by tablet a68badb3b5644ed8b007b2b04bc4f6e1) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.903503 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_889601b1d99641d8b8420888cf849ec1" } modify_external_catalogs: true
I20240507 08:12:11.908401 6880 raft_consensus.cc:482] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.908547 6880 raft_consensus.cc:504] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.909126 6880 leader_election.cc:288] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.909744 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.910439 5378 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.910638 6513 tablet_replica.cc:330] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:11.911881 6513 raft_consensus.cc:2229] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:11.912472 6513 raft_consensus.cc:2258] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.913172 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:11.913669 5377 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.913766 5377 leader_election.cc:302] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.914314 6880 raft_consensus.cc:2726] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.917169 5839 tablet_bootstrap.cc:492] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:11.917333 5839 ts_tablet_manager.cc:1367] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.089s user 0.000s sys 0.003s
I20240507 08:12:11.918130 5839 raft_consensus.cc:348] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.918360 5839 raft_consensus.cc:374] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.918452 5839 raft_consensus.cc:729] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.918884 5839 consensus_queue.cc:260] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.922292 5843 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:11.922475 5843 ts_tablet_manager.cc:1367] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.010s
I20240507 08:12:11.923134 5843 raft_consensus.cc:348] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.923368 5843 raft_consensus.cc:374] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.923492 5843 raft_consensus.cc:729] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.923743 5843 consensus_queue.cc:260] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.923852 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 0118ef28b24641d68cf24ce9df966243 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fce98f472ef46589dbea6b6aefa6572) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.924451 5843 ts_tablet_manager.cc:1398] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:11.925148 5839 ts_tablet_manager.cc:1398] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240507 08:12:11.927860 5843 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:11.928668 5839 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:11.929241 7079 ts_tablet_manager.cc:1875] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.930404 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.932179 6513 ts_tablet_manager.cc:1875] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:11.932869 6880 raft_consensus.cc:482] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.932988 6880 raft_consensus.cc:504] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.933595 6880 leader_election.cc:288] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.934325 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.934882 5378 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.936352 5843 tablet_bootstrap.cc:654] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.935426 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:11.937285 5840 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:11.937816 5719 raft_consensus.cc:2454] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:11.938732 5840 ts_tablet_manager.cc:1367] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.111s user 0.009s sys 0.004s
I20240507 08:12:11.939577 5377 leader_election.cc:302] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.939539 5840 raft_consensus.cc:348] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.939759 5840 raft_consensus.cc:374] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:11.939844 5840 raft_consensus.cc:729] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:11.940107 6880 raft_consensus.cc:2781] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:11.940182 6880 raft_consensus.cc:482] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:11.940246 6880 raft_consensus.cc:3037] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:11.940174 5840 consensus_queue.cc:260] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.942766 6880 raft_consensus.cc:504] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.943514 6880 leader_election.cc:288] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.943902 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:11.943943 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "34adee00b529424b9f274c61ce73b0bc" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:11.944160 5719 raft_consensus.cc:3037] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:11.944432 5378 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.945099 5840 ts_tablet_manager.cc:1398] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:11.945427 5840 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:11.946974 5840 tablet_bootstrap.cc:654] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.947170 5719 raft_consensus.cc:2454] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:11.949198 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d0aa0582569b459bbed495999da97539"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.949806 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0aa0582569b459bbed495999da97539 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.949887 5839 tablet_bootstrap.cc:654] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:11.950446 5841 ts_tablet_manager.cc:1888] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:11.950558 5841 log.cc:1196] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:12:11.950917 5841 ts_tablet_manager.cc:1909] T 7ba2e8d1e01047fd9cc1c192e9628750 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:11.949154 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ebe94534f9de4a8eaf2d035236c832dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:11.951738 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 838) took 5427 ms. Trace:
I20240507 08:12:11.951833 5841 rpcz_store.cc:276] 0507 08:12:06.523954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.526781 (+ 2827us) service_pool.cc:225] Handling call
0507 08:12:11.890282 (+5363501us) ts_tablet_manager.cc:1120] Deleting tablet 7ba2e8d1e01047fd9cc1c192e9628750
0507 08:12:11.890297 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.921016 (+ 30719us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.950343 (+ 29327us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:11.951715 (+ 1372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5357985}
W20240507 08:12:11.952081 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebe94534f9de4a8eaf2d035236c832dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.952185 5841 tablet_replica.cc:330] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:11.952375 5841 raft_consensus.cc:2229] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:11.952526 5841 raft_consensus.cc:2258] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:11.952716 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6e0d718bf25e458d8090be71389a3c89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:11.953087 5841 ts_tablet_manager.cc:1875] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:11.953136 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e0d718bf25e458d8090be71389a3c89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.953240 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7ba2e8d1e01047fd9cc1c192e9628750 (table default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]) successfully deleted
I20240507 08:12:11.953974 5377 leader_election.cc:302] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.954797 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3dbb85e2d6814888a9680e5bd557124d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.955538 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dbb85e2d6814888a9680e5bd557124d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.956904 6880 raft_consensus.cc:2781] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:11.957171 6880 raft_consensus.cc:686] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:11.957453 6880 consensus_queue.cc:237] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.960283 7107 raft_consensus.cc:482] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:11.960399 7107 raft_consensus.cc:504] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:11.960989 7107 leader_election.cc:288] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:11.961679 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:11.962146 5377 leader_election.cc:341] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.961747 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:11.962877 5378 leader_election.cc:341] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:11.963094 5378 leader_election.cc:302] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:11.858477 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a69f8756b5934280bf92b31a383e0f88" } modify_external_catalogs: true
I20240507 08:12:11.967887 7107 raft_consensus.cc:2726] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:11.971797 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 37a23d112fcd4ea1b0c2f86133e82e32 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:11.982419 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet dc74342c8aee4017a212aa88a7f3dc06
I20240507 08:12:11.983232 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:11.983551 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:11.985076 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet dc74342c8aee4017a212aa88a7f3dc06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:11.990746 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1ddacc4556034a0892446b12705d92cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:11.991346 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ddacc4556034a0892446b12705d92cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:11.992693 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_50b030da7ecf4a98888919bc423e74a0" } new_table_name: "default.Table_03fad5ee76d246eaa9e15ae1eba26791" modify_external_catalogs: true
I20240507 08:12:11.999701 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:12.007120 5841 ts_tablet_manager.cc:1888] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:12.007866 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_adbdf44307174a5b8a7a916c5fc6c5bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.008498 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adbdf44307174a5b8a7a916c5fc6c5bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.008667 5841 log.cc:1196] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f294baf04bf74dfe8500cb8c1452d8b3
I20240507 08:12:12.008980 5841 ts_tablet_manager.cc:1909] T f294baf04bf74dfe8500cb8c1452d8b3 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:12.009346 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 839) took 5449 ms. Trace:
I20240507 08:12:12.009425 5841 rpcz_store.cc:276] 0507 08:12:06.559740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.559807 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:11.952137 (+5392330us) ts_tablet_manager.cc:1120] Deleting tablet f294baf04bf74dfe8500cb8c1452d8b3
0507 08:12:11.952149 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.983182 (+ 31033us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.006980 (+ 23798us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.009321 (+ 2341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5392138}
I20240507 08:12:12.010008 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f294baf04bf74dfe8500cb8c1452d8b3 (table default.Table_6d8668c1f9b94e15b7b7603a49f11a7d [id=6f1cedff6040494bb0a9bfe571648812]) successfully deleted
W20240507 08:12:12.013322 5378 consensus_peers.cc:485] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.019323 5841 tablet_replica.cc:330] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.019682 5839 tablet_bootstrap.cc:492] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.019836 5839 ts_tablet_manager.cc:1367] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.004s
I20240507 08:12:12.020156 5841 raft_consensus.cc:2229] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:12.020552 5839 raft_consensus.cc:348] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.020750 5839 raft_consensus.cc:374] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.020814 5839 raft_consensus.cc:729] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.021564 5841 raft_consensus.cc:2258] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.023046 5841 ts_tablet_manager.cc:1875] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.024809 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_95bd32ae1ac34b0f8588ec7fa4c8e50a" } new_table_name: "default.Table_27c86e8536c344a7bf06b2f6b13b3082" modify_external_catalogs: true
I20240507 08:12:12.035040 7079 ts_tablet_manager.cc:1888] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:12.035178 7079 log.cc:1196] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/a3e58ba8c26d4f17af24eb379aca1f60
I20240507 08:12:12.035488 7079 ts_tablet_manager.cc:1909] T a3e58ba8c26d4f17af24eb379aca1f60 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:12.036032 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet a3e58ba8c26d4f17af24eb379aca1f60 (table default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]) successfully deleted
I20240507 08:12:12.041010 7101 consensus_queue.cc:1035] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.047631 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8e2edbe2bb52425bbde255742f34f784"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.048543 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e2edbe2bb52425bbde255742f34f784 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.048779 7055 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:12.054584 5843 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.055999 5843 ts_tablet_manager.cc:1367] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.128s user 0.013s sys 0.006s
I20240507 08:12:12.057075 5843 raft_consensus.cc:348] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.059008 5843 raft_consensus.cc:374] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.059306 5843 raft_consensus.cc:729] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.059857 5843 consensus_queue.cc:260] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.060740 7245 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:12.061224 5843 ts_tablet_manager.cc:1398] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:12.061822 5843 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.066264 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_200c77e7e235497c952d6a523e6a30e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.066819 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_200c77e7e235497c952d6a523e6a30e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.066958 7214 raft_consensus.cc:482] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.067092 7214 raft_consensus.cc:504] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.067708 7214 leader_election.cc:288] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:12.068353 5517 leader_election.cc:341] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 97c978ec8bca4aa5b2ffaf22c3290747
W20240507 08:12:12.069696 5520 leader_election.cc:341] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 97c978ec8bca4aa5b2ffaf22c3290747
I20240507 08:12:12.069804 5520 leader_election.cc:302] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:12.070225 7214 raft_consensus.cc:2726] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.069705 5839 consensus_queue.cc:260] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.079744 7100 raft_consensus.cc:482] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.079988 5839 ts_tablet_manager.cc:1398] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.060s user 0.000s sys 0.002s
I20240507 08:12:12.079928 7100 raft_consensus.cc:504] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.080587 7100 leader_election.cc:288] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.080627 5839 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.080761 7100 consensus_queue.cc:1035] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.081117 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.081306 5443 raft_consensus.cc:2454] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.082054 5656 leader_election.cc:302] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:12.082329 6675 raft_consensus.cc:2781] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:12.082450 6675 raft_consensus.cc:482] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:12.082551 6675 raft_consensus.cc:3037] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.086084 7079 tablet_replica.cc:330] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:12.086676 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:12.085460 6675 raft_consensus.cc:504] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.087353 6675 leader_election.cc:288] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.088042 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:12.088250 5439 raft_consensus.cc:3037] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.089836 7079 raft_consensus.cc:2229] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.090893 7079 raft_consensus.cc:2258] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.091329 5840 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.091320 5439 raft_consensus.cc:2454] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:12.091614 5840 ts_tablet_manager.cc:1367] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.146s user 0.010s sys 0.010s
I20240507 08:12:12.092145 5656 leader_election.cc:302] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:12.092535 5840 raft_consensus.cc:348] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.092729 5840 raft_consensus.cc:374] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.092828 5840 raft_consensus.cc:729] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.092815 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:12.093333 5654 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.094348 6675 raft_consensus.cc:2781] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:12.094535 6675 raft_consensus.cc:686] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:12.094823 6675 consensus_queue.cc:237] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.092504 6513 ts_tablet_manager.cc:1888] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.095834 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "049483b0533642f2b46d8fa0e9b123cc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.096347 5654 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.096189 6513 log.cc:1196] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/0118ef28b24641d68cf24ce9df966243
I20240507 08:12:12.097397 7079 ts_tablet_manager.cc:1875] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.098735 5839 tablet_bootstrap.cc:654] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.099486 6513 ts_tablet_manager.cc:1909] T 0118ef28b24641d68cf24ce9df966243 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:12.102753 5843 tablet_bootstrap.cc:654] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.093474 5840 consensus_queue.cc:260] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.109555 5840 ts_tablet_manager.cc:1398] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
W20240507 08:12:12.109663 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce73981ca280473eaa2941447313d7a7 consensus state has no leader
I20240507 08:12:12.110280 5840 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:12.111980 5840 tablet_bootstrap.cc:654] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.116916 5843 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.117166 5843 ts_tablet_manager.cc:1367] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.002s
I20240507 08:12:12.120281 5843 raft_consensus.cc:348] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.120608 5843 raft_consensus.cc:374] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.120723 5843 raft_consensus.cc:729] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.121012 5843 consensus_queue.cc:260] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.121695 5843 ts_tablet_manager.cc:1398] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:12.122776 5840 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.122968 5840 ts_tablet_manager.cc:1367] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20240507 08:12:12.123708 5840 raft_consensus.cc:348] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.124122 5840 raft_consensus.cc:374] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.124256 5840 raft_consensus.cc:729] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.124392 5841 ts_tablet_manager.cc:1888] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.124641 5840 consensus_queue.cc:260] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.132270 5840 ts_tablet_manager.cc:1398] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:12.135381 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_63548a17f66844648a2d71fb0c7d7175" } new_table_name: "default.Table_61a1b114b4fb46fb9923fc7426928be6" modify_external_catalogs: true
I20240507 08:12:12.136054 7214 raft_consensus.cc:482] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.136212 7214 raft_consensus.cc:504] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.136543 5840 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:12.136844 7214 leader_election.cc:288] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:12.138617 5520 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f294baf04bf74dfe8500cb8c1452d8b3
W20240507 08:12:12.138943 5517 leader_election.cc:341] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: f294baf04bf74dfe8500cb8c1452d8b3
I20240507 08:12:12.139077 5517 leader_election.cc:302] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:12.139691 7214 raft_consensus.cc:2726] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.140146 5840 tablet_bootstrap.cc:654] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.146014 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
W20240507 08:12:12.146925 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42809e2f45e9466594377b41ae920f3b consensus state has no leader
I20240507 08:12:12.149835 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ad88688ccc4f46edaafaf3b8c6e3f8ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.150398 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad88688ccc4f46edaafaf3b8c6e3f8ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:12.157380 5654 consensus_peers.cc:485] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.153038 7107 raft_consensus.cc:482] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.158234 7107 raft_consensus.cc:504] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.159077 7107 leader_election.cc:288] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.159520 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.159788 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:12.160085 5378 leader_election.cc:341] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:12.160207 5377 leader_election.cc:341] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.160337 5377 leader_election.cc:302] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.160838 7107 raft_consensus.cc:2726] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.158213 5841 log.cc:1196] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4de7053cfaea4a54a402c9d826613c05
I20240507 08:12:12.161664 5841 ts_tablet_manager.cc:1909] T 4de7053cfaea4a54a402c9d826613c05 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:12.164688 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 915) took 3542 ms. Trace:
I20240507 08:12:12.164809 6513 rpcz_store.cc:276] 0507 08:12:08.622261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955519 (+333258us) service_pool.cc:225] Handling call
0507 08:12:11.881105 (+2925586us) ts_tablet_manager.cc:1120] Deleting tablet 0118ef28b24641d68cf24ce9df966243
0507 08:12:11.910494 (+ 29389us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:11.975302 (+ 64808us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.091326 (+116024us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.164619 (+ 73293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2925465}
I20240507 08:12:12.165241 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 0118ef28b24641d68cf24ce9df966243 (table default.Table_727fd7e435fd40368f63b993db76af91 [id=786a00558b0f42cd9700cd3200dfe1ac]) successfully deleted
I20240507 08:12:12.171742 7248 tablet.cc:1722] T d0ea74702c4b4514a31f0fad7e7a2f5b P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:12.165010 5843 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.173852 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet ba7c1aaf55d74c979ec1fc0a57f15f44 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:12.174186 5843 tablet_bootstrap.cc:654] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.175735 7055 tablet.cc:1722] T ed2604a090c94c78bd4f7a1dea6f81b7 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:12.176582 6513 tablet_replica.cc:330] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:12.176829 6513 raft_consensus.cc:2229] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.177059 6513 raft_consensus.cc:2258] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.177672 6513 ts_tablet_manager.cc:1875] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.182709 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 840) took 5513 ms. Trace:
I20240507 08:12:12.183310 5841 rpcz_store.cc:276] 0507 08:12:06.669272 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.672000 (+ 2728us) service_pool.cc:225] Handling call
0507 08:12:12.019257 (+5347257us) ts_tablet_manager.cc:1120] Deleting tablet 4de7053cfaea4a54a402c9d826613c05
0507 08:12:12.019270 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.066622 (+ 47352us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.124277 (+ 57655us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.182681 (+ 58404us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5347045}
I20240507 08:12:12.184044 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4de7053cfaea4a54a402c9d826613c05 (table default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]) successfully deleted
I20240507 08:12:12.184489 5841 tablet_replica.cc:330] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.184804 5841 raft_consensus.cc:2229] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.185137 5841 raft_consensus.cc:2258] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.185894 5841 ts_tablet_manager.cc:1875] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.193997 5843 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.194229 5843 ts_tablet_manager.cc:1367] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20240507 08:12:12.195194 5843 raft_consensus.cc:348] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.195511 5843 raft_consensus.cc:374] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.195617 5843 raft_consensus.cc:729] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.195930 5843 consensus_queue.cc:260] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.199429 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_74303359aed849ecb2e92ecb0fe8914d" } new_table_name: "default.Table_47188cab4dee4904b2a5099a06b2a652" modify_external_catalogs: true
I20240507 08:12:12.201716 5843 ts_tablet_manager.cc:1398] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:12.202119 5843 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.202756 5839 tablet_bootstrap.cc:492] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.202903 5839 ts_tablet_manager.cc:1367] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.122s user 0.004s sys 0.000s
I20240507 08:12:12.203716 5843 tablet_bootstrap.cc:654] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.203617 5839 raft_consensus.cc:348] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.203817 5839 raft_consensus.cc:374] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.203879 5839 raft_consensus.cc:729] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.204128 5839 consensus_queue.cc:260] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.208746 7248 tablet.cc:1722] T ed2604a090c94c78bd4f7a1dea6f81b7 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:12.211226 5839 ts_tablet_manager.cc:1398] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:12:12.213351 7079 ts_tablet_manager.cc:1888] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.214008 6675 consensus_queue.cc:1035] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.219105 5839 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.214109 7079 log.cc:1196] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/7b4b84ab3f274185a2b838ebb969ab52
I20240507 08:12:12.223405 7245 tablet.cc:1722] T ed2604a090c94c78bd4f7a1dea6f81b7 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:12.228214 5843 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.228385 5843 ts_tablet_manager.cc:1367] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20240507 08:12:12.229137 5843 raft_consensus.cc:348] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.229432 5843 raft_consensus.cc:374] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.229553 5843 raft_consensus.cc:729] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.229911 5843 consensus_queue.cc:260] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.230592 5843 ts_tablet_manager.cc:1398] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:12.230945 5843 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.232304 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_ca857f7ec80945ff80de395cbcd59f8d [id=b3fb019c81734b0f8108f607f4b15972] alter complete (version 3)
W20240507 08:12:12.232801 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1d5627f71694fb68a15fb186c9068ce consensus state has no leader
I20240507 08:12:12.233060 7107 raft_consensus.cc:482] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.233175 7107 raft_consensus.cc:504] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.233189 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d3257e1057da421d901ffdb965228250" } modify_external_catalogs: true
I20240507 08:12:12.233393 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d3257e1057da421d901ffdb965228250" } modify_external_catalogs: true
I20240507 08:12:12.233785 7107 leader_election.cc:288] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.233879 7079 ts_tablet_manager.cc:1909] T 7b4b84ab3f274185a2b838ebb969ab52 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:12.234602 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:12.235342 7079 tablet_replica.cc:330] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:12.235486 5377 leader_election.cc:341] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.235771 7079 raft_consensus.cc:2229] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.235883 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.236027 5843 tablet_bootstrap.cc:654] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.236219 7079 raft_consensus.cc:2258] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:12.236497 5378 leader_election.cc:341] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.236629 5378 leader_election.cc:302] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.237041 7107 raft_consensus.cc:2726] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.240131 5839 tablet_bootstrap.cc:654] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.240422 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7d1b03d5425d44c695c58cb9cfa6f68c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.240942 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d1b03d5425d44c695c58cb9cfa6f68c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.234691 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 7b4b84ab3f274185a2b838ebb969ab52 (table default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]) successfully deleted
W20240507 08:12:12.242165 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d6f8f4205f1f48abaf4fcd7113ad44a8 because the tablet was not found. No further retry: Not found: Tablet not found: d6f8f4205f1f48abaf4fcd7113ad44a8
I20240507 08:12:12.248557 5841 ts_tablet_manager.cc:1888] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.254186 5841 log.cc:1196] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:12:12.254823 5841 ts_tablet_manager.cc:1909] T 3b5e3d23bbb6439a87b303dff1b44a1d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:12.257247 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2572985bc2444dc7b051f8fe008e3ad4 consensus state has no leader
I20240507 08:12:12.257704 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_855c7a629b1c4b48a6859a9d4d812be6" } modify_external_catalogs: true
I20240507 08:12:12.257918 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_855c7a629b1c4b48a6859a9d4d812be6" } modify_external_catalogs: true
I20240507 08:12:12.258085 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 841) took 5575 ms. Trace:
I20240507 08:12:12.258193 5841 rpcz_store.cc:276] 0507 08:12:06.683104 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.683156 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:12.184403 (+5501247us) ts_tablet_manager.cc:1120] Deleting tablet 3b5e3d23bbb6439a87b303dff1b44a1d
0507 08:12:12.184438 (+ 35us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.227565 (+ 43127us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.248426 (+ 20861us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.258052 (+ 9626us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5501007}
I20240507 08:12:12.260389 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3b5e3d23bbb6439a87b303dff1b44a1d (table default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]) successfully deleted
I20240507 08:12:12.260834 5841 tablet_replica.cc:330] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.261142 5841 raft_consensus.cc:2229] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.261390 5841 raft_consensus.cc:2258] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.262363 5841 ts_tablet_manager.cc:1875] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.263721 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ba7c1aaf55d74c979ec1fc0a57f15f44
I20240507 08:12:12.264667 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f9d270dcfe84128a31fe48d5f2be090) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.264922 5261 catalog_manager.cc:6554] ReplaceTablet: tablet ba7c1aaf55d74c979ec1fc0a57f15f44 deleted and replaced by tablet 8f9d270dcfe84128a31fe48d5f2be090
I20240507 08:12:12.265334 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bc3afc777e354e2e8a8ae7266a761d73" } new_table_name: "default.Table_ec36ee6c38ab4cd29001e2a9488665c8" modify_external_catalogs: true
I20240507 08:12:12.267133 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f9d270dcfe84128a31fe48d5f2be090) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.267473 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f9d270dcfe84128a31fe48d5f2be090) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:12.268910 5654 consensus_peers.cc:485] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.270707 6675 consensus_queue.cc:1035] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.270958 7079 ts_tablet_manager.cc:1875] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:12.279505 5654 consensus_peers.cc:485] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.282156 5843 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.282420 5843 ts_tablet_manager.cc:1367] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.003s
I20240507 08:12:12.283003 6513 ts_tablet_manager.cc:1888] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.283214 6513 log.cc:1196] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4e63c3f3ecc84913ad05864be450e9b0
I20240507 08:12:12.283376 5843 raft_consensus.cc:348] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.283722 6513 ts_tablet_manager.cc:1909] T 4e63c3f3ecc84913ad05864be450e9b0 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:12.284579 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 916) took 3662 ms. Trace:
I20240507 08:12:12.284857 6513 rpcz_store.cc:276] 0507 08:12:08.622347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955615 (+333268us) service_pool.cc:225] Handling call
0507 08:12:12.176478 (+3220863us) ts_tablet_manager.cc:1120] Deleting tablet 4e63c3f3ecc84913ad05864be450e9b0
0507 08:12:12.176490 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.204184 (+ 27694us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.282855 (+ 78671us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.284544 (+ 1689us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3220751}
I20240507 08:12:12.285275 5843 raft_consensus.cc:374] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.285377 5843 raft_consensus.cc:729] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.285542 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4e63c3f3ecc84913ad05864be450e9b0 (table default.Table_3da110b2f7014fdca83ebeeb4c70c3d1 [id=352055be23d04485b3896bd169ea07ba]) successfully deleted
I20240507 08:12:12.285650 5843 consensus_queue.cc:260] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.286365 5843 ts_tablet_manager.cc:1398] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:12.286682 5843 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.286927 5840 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.288641 5843 tablet_bootstrap.cc:654] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.290572 5840 ts_tablet_manager.cc:1367] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.154s user 0.007s sys 0.023s
I20240507 08:12:12.292018 5840 raft_consensus.cc:348] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.293171 5840 raft_consensus.cc:374] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.293371 5840 raft_consensus.cc:729] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.293766 5840 consensus_queue.cc:260] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.294821 5840 ts_tablet_manager.cc:1398] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:12.295938 5840 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:12.298265 5840 tablet_bootstrap.cc:654] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.301367 5843 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.302048 5843 ts_tablet_manager.cc:1367] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20240507 08:12:12.302161 7107 raft_consensus.cc:482] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.302417 7107 raft_consensus.cc:504] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.303081 7107 leader_election.cc:288] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.306079 5843 raft_consensus.cc:348] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.307370 5843 raft_consensus.cc:374] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.307509 5843 raft_consensus.cc:729] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.307535 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:12.308037 7214 raft_consensus.cc:482] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:12.308178 5377 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:12.308241 5654 consensus_peers.cc:485] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.307904 5843 consensus_queue.cc:260] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.308180 7214 raft_consensus.cc:504] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.308476 7101 raft_consensus.cc:482] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.308599 7101 raft_consensus.cc:504] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.308789 5843 ts_tablet_manager.cc:1398] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:12.308832 7214 leader_election.cc:288] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.309069 5843 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.309342 7101 leader_election.cc:288] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.309849 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.310420 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.310662 5439 raft_consensus.cc:2454] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.310882 5843 tablet_bootstrap.cc:654] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:12.311020 5654 leader_election.cc:341] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.310446 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.311288 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.312065 5656 leader_election.cc:302] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.312191 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5fb384ed8b7546e39c39acc6facec9f5" } modify_external_catalogs: true
I20240507 08:12:12.312187 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e17502bb4ac64f6ca9c77a3b00ab9f4a" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:12.312376 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5fb384ed8b7546e39c39acc6facec9f5" } modify_external_catalogs: true
W20240507 08:12:12.313663 5378 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.313782 5378 leader_election.cc:302] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.314011 7107 raft_consensus.cc:2726] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.314530 7100 raft_consensus.cc:2781] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:12.314567 5517 leader_election.cc:302] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:12.314661 7100 raft_consensus.cc:482] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:12.314740 7100 raft_consensus.cc:3037] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.314932 7214 raft_consensus.cc:2726] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.317369 7100 raft_consensus.cc:504] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.318274 7100 leader_election.cc:288] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.319379 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_f517b84d78b54bc39275ee1d49b79bfd [id=26bfe0dbbeca415eb1900ef1ab5bd316] alter complete (version 1)
I20240507 08:12:12.319962 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:12.320583 5443 raft_consensus.cc:3037] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.320809 5839 tablet_bootstrap.cc:492] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.320941 5839 ts_tablet_manager.cc:1367] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.005s
I20240507 08:12:12.321712 5839 raft_consensus.cc:348] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.321892 5839 raft_consensus.cc:374] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.321941 5839 raft_consensus.cc:729] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.322101 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43644b4405f44bfd9a1afcaedbd68db5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:12.322425 7107 consensus_queue.cc:1035] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:12.322890 5654 leader_election.cc:341] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.323127 5839 consensus_queue.cc:260] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.326025 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7079f1efc8a149a9b81f0b7667492b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.326591 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7079f1efc8a149a9b81f0b7667492b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.328670 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet f294baf04bf74dfe8500cb8c1452d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.328910 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 4de7053cfaea4a54a402c9d826613c05 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.332084 5839 ts_tablet_manager.cc:1398] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:12.332540 5839 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.336536 5443 raft_consensus.cc:2454] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:12.337715 5656 leader_election.cc:302] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.338356 7100 raft_consensus.cc:2781] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:12.338618 7100 raft_consensus.cc:686] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:12.338960 7100 consensus_queue.cc:237] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.340348 7107 raft_consensus.cc:482] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.340541 7107 raft_consensus.cc:504] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.341192 7107 leader_election.cc:288] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.342700 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:12.342720 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.343152 5377 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:12.343277 5378 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.343391 5378 leader_election.cc:302] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.343842 7107 raft_consensus.cc:2726] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.346094 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bba5a7409437414987e560b41b2c8875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.346680 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bba5a7409437414987e560b41b2c8875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.346710 5841 ts_tablet_manager.cc:1888] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:12.346851 5841 log.cc:1196] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/150efae7834740b798af86ae1dcfe445
I20240507 08:12:12.347180 5841 ts_tablet_manager.cc:1909] T 150efae7834740b798af86ae1dcfe445 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:12.352140 5839 tablet_bootstrap.cc:654] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.305119 6513 tablet_replica.cc:330] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:12.353237 6513 raft_consensus.cc:2229] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.354598 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 842) took 5649 ms. Trace:
I20240507 08:12:12.355409 6513 raft_consensus.cc:2258] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.355326 5841 rpcz_store.cc:276] 0507 08:12:06.705055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.705122 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:12.260771 (+5555649us) ts_tablet_manager.cc:1120] Deleting tablet 150efae7834740b798af86ae1dcfe445
0507 08:12:12.260785 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.291312 (+ 30527us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.346557 (+ 55245us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.354575 (+ 8018us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5555394}
I20240507 08:12:12.356035 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 150efae7834740b798af86ae1dcfe445 (table default.Table_1d73a8ee5a4848628ed2472bdc0b084b [id=d95a0f67e809437da87ad837fbf8f2a3]) successfully deleted
I20240507 08:12:12.356832 6513 ts_tablet_manager.cc:1875] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.360319 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 843) took 5644 ms. Trace:
I20240507 08:12:12.361373 5841 rpcz_store.cc:276] 0507 08:12:06.715374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.715762 (+ 388us) service_pool.cc:225] Handling call
0507 08:12:12.356694 (+5640932us) ts_tablet_manager.cc:1120] Deleting tablet 242ea35c61404a91b1211c5ad9207728
0507 08:12:12.360154 (+ 3460us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.360292 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5640745}
W20240507 08:12:12.362686 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 242ea35c61404a91b1211c5ad9207728 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 242ea35c61404a91b1211c5ad9207728 already in progress: creating tablet
I20240507 08:12:12.362579 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e9828e7d20ab4c959f6963420d4fa85e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:12.362895 5840 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.363056 5840 ts_tablet_manager.cc:1367] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
W20240507 08:12:12.363216 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9828e7d20ab4c959f6963420d4fa85e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.363377 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 844) took 5493 ms. Trace:
I20240507 08:12:12.363802 5841 rpcz_store.cc:276] 0507 08:12:06.870044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.870114 (+ 70us) service_pool.cc:225] Handling call
0507 08:12:12.363234 (+5493120us) ts_tablet_manager.cc:1120] Deleting tablet 96d7d8bcb635416693395b1f9ad27e2f
0507 08:12:12.363247 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.363361 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5492853}
W20240507 08:12:12.364454 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 96d7d8bcb635416693395b1f9ad27e2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96d7d8bcb635416693395b1f9ad27e2f already in progress: creating tablet
I20240507 08:12:12.364885 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 847) took 5373 ms. Trace:
I20240507 08:12:12.365242 5841 rpcz_store.cc:276] 0507 08:12:06.991620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:06.995623 (+ 4003us) service_pool.cc:225] Handling call
0507 08:12:12.364786 (+5369163us) ts_tablet_manager.cc:1120] Deleting tablet abd47fb41bd14411bb44cc0f25e8cac4
0507 08:12:12.364795 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.364870 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5365485}
W20240507 08:12:12.366026 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet abd47fb41bd14411bb44cc0f25e8cac4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet abd47fb41bd14411bb44cc0f25e8cac4 already in progress: creating tablet
I20240507 08:12:12.366441 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 855) took 5319 ms. Trace:
I20240507 08:12:12.366783 5841 rpcz_store.cc:276] 0507 08:12:07.046628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.049622 (+ 2994us) service_pool.cc:225] Handling call
0507 08:12:12.366342 (+5316720us) ts_tablet_manager.cc:1120] Deleting tablet aa485207d0f140c99a9386cb4002c480
0507 08:12:12.366351 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.366426 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5316540}
W20240507 08:12:12.367383 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet aa485207d0f140c99a9386cb4002c480 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa485207d0f140c99a9386cb4002c480 already in progress: creating tablet
I20240507 08:12:12.367780 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 867) took 5178 ms. Trace:
I20240507 08:12:12.368139 5841 rpcz_store.cc:276] 0507 08:12:07.189545 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.189630 (+ 85us) service_pool.cc:225] Handling call
0507 08:12:12.367683 (+5178053us) ts_tablet_manager.cc:1120] Deleting tablet b60ec15a83934999871425ef9a64ec72
0507 08:12:12.367692 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.367766 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5177792}
W20240507 08:12:12.368746 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b60ec15a83934999871425ef9a64ec72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b60ec15a83934999871425ef9a64ec72 already in progress: creating tablet
I20240507 08:12:12.369139 5841 tablet_replica.cc:330] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.369609 5841 raft_consensus.cc:2229] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.370584 5841 raft_consensus.cc:2258] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.372248 5841 ts_tablet_manager.cc:1875] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.355171 7079 ts_tablet_manager.cc:1888] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:12.372738 7079 log.cc:1196] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/97c978ec8bca4aa5b2ffaf22c3290747
I20240507 08:12:12.373086 7079 ts_tablet_manager.cc:1909] T 97c978ec8bca4aa5b2ffaf22c3290747 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:12.363457 5656 consensus_peers.cc:485] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3e58ba8c26d4f17af24eb379aca1f60. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.374742 7248 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:12.375272 5840 raft_consensus.cc:348] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.375641 5840 raft_consensus.cc:374] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.375784 5840 raft_consensus.cc:729] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.376206 5840 consensus_queue.cc:260] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.377027 5840 ts_tablet_manager.cc:1398] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:12.377422 5840 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:12.373507 5839 tablet_bootstrap.cc:492] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.377799 5839 ts_tablet_manager.cc:1367] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.001s
I20240507 08:12:12.375764 7055 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:12.378592 5839 raft_consensus.cc:348] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.378772 5839 raft_consensus.cc:374] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.378827 5839 raft_consensus.cc:729] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.379025 5839 consensus_queue.cc:260] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.381464 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d7b00c2cd3f84b2ba1f5c18b18d93bb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.382249 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7b00c2cd3f84b2ba1f5c18b18d93bb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.383865 5839 ts_tablet_manager.cc:1398] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:12.384192 5839 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.386129 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 97c978ec8bca4aa5b2ffaf22c3290747 (table default.Table_6256978b02b54f189b2c9d7b524d6f28 [id=d738d9196bbe46fa80802bf9f5d4ea93]) successfully deleted
W20240507 08:12:12.386912 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e3783fc7fe434696898ba14ab77a9058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3783fc7fe434696898ba14ab77a9058 already in progress: creating tablet
W20240507 08:12:12.387210 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aafd1dc2eaf84390b072f01a9e76232d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafd1dc2eaf84390b072f01a9e76232d already in progress: creating tablet
W20240507 08:12:12.387451 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 323b96ae7d1e471998a39f3be0857ce7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323b96ae7d1e471998a39f3be0857ce7 already in progress: creating tablet
W20240507 08:12:12.387679 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cacfe980b2aa4a60ad05bbe102a94266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cacfe980b2aa4a60ad05bbe102a94266 already in progress: creating tablet
I20240507 08:12:12.387842 7079 tablet_replica.cc:330] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:12.387898 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6ad4ed7878454fea930b3982bcb6eed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ad4ed7878454fea930b3982bcb6eed4 already in progress: creating tablet
I20240507 08:12:12.388005 7079 raft_consensus.cc:2229] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.388139 7079 raft_consensus.cc:2258] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.390811 5840 tablet_bootstrap.cc:654] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.391558 5839 tablet_bootstrap.cc:654] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.369163 6513 ts_tablet_manager.cc:1888] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.395066 6513 log.cc:1196] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4de7053cfaea4a54a402c9d826613c05
I20240507 08:12:12.396319 6513 ts_tablet_manager.cc:1909] T 4de7053cfaea4a54a402c9d826613c05 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:12.396203 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8f309bfc29fa40b39a9b836aff0d30ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.396721 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f309bfc29fa40b39a9b836aff0d30ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.397246 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet c43f3de489904f2ba2a637d408619bfc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.398447 5839 tablet_bootstrap.cc:492] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.399031 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 917) took 3776 ms. Trace:
I20240507 08:12:12.408779 5841 ts_tablet_manager.cc:1888] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.417964 5839 ts_tablet_manager.cc:1367] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20240507 08:12:12.418265 5841 log.cc:1196] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/44ec0526e1914221bc22b3e75ea01e0d
I20240507 08:12:12.393406 5843 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:12.418542 5843 ts_tablet_manager.cc:1367] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.110s user 0.004s sys 0.000s
I20240507 08:12:12.419486 5843 raft_consensus.cc:348] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.419787 5843 raft_consensus.cc:374] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.419924 5843 raft_consensus.cc:729] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.419898 6513 rpcz_store.cc:276] 0507 08:12:08.622441 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955709 (+333268us) service_pool.cc:225] Handling call
0507 08:12:12.304541 (+3348832us) ts_tablet_manager.cc:1120] Deleting tablet 4de7053cfaea4a54a402c9d826613c05
0507 08:12:12.304551 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.365820 (+ 61269us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.369048 (+ 3228us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.398974 (+ 29926us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3348719}
I20240507 08:12:12.420279 5843 consensus_queue.cc:260] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.421103 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4de7053cfaea4a54a402c9d826613c05 (table default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]) successfully deleted
I20240507 08:12:12.421700 6513 tablet_replica.cc:330] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:12.422745 6513 raft_consensus.cc:2229] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:12.423112 6513 raft_consensus.cc:2258] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.423522 5839 raft_consensus.cc:348] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.423719 5839 raft_consensus.cc:374] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.423769 5839 raft_consensus.cc:729] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.423983 5839 consensus_queue.cc:260] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.425801 6513 ts_tablet_manager.cc:1875] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.426075 7245 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:12.428226 5843 ts_tablet_manager.cc:1398] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240507 08:12:12.428246 7079 ts_tablet_manager.cc:1875] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.433873 5843 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:12.435089 5839 ts_tablet_manager.cc:1398] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240507 08:12:12.435708 5841 ts_tablet_manager.cc:1909] T 44ec0526e1914221bc22b3e75ea01e0d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:12.436266 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 868) took 5103 ms. Trace:
I20240507 08:12:12.436964 5841 rpcz_store.cc:276] 0507 08:12:07.332745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.332819 (+ 74us) service_pool.cc:225] Handling call
0507 08:12:12.369097 (+5036278us) ts_tablet_manager.cc:1120] Deleting tablet 44ec0526e1914221bc22b3e75ea01e0d
0507 08:12:12.369105 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.405470 (+ 36365us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.408622 (+ 3152us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.436209 (+ 27587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036014}
I20240507 08:12:12.437983 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 44ec0526e1914221bc22b3e75ea01e0d (table default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]) successfully deleted
I20240507 08:12:12.436882 5843 tablet_bootstrap.cc:654] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.454241 7101 raft_consensus.cc:482] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.456097 7101 raft_consensus.cc:504] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.456684 7101 leader_election.cc:288] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:12.455850 5654 consensus_peers.cc:485] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.457432 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.462680 5654 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.463106 5839 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.464807 5839 tablet_bootstrap.cc:654] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.466302 5584 raft_consensus.cc:3037] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:12.471174 5654 consensus_peers.cc:485] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.474166 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 869) took 5078 ms. Trace:
I20240507 08:12:12.474303 5841 rpcz_store.cc:276] 0507 08:12:07.395946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.396155 (+ 209us) service_pool.cc:225] Handling call
0507 08:12:12.438336 (+5042181us) ts_tablet_manager.cc:1120] Deleting tablet 5f0b399a2d074f4ca3626ff17bd5518e
0507 08:12:12.438344 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.474108 (+ 35764us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5041943}
I20240507 08:12:12.476208 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.479851 5841 tablet_replica.cc:330] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.480126 5841 raft_consensus.cc:2229] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.480332 5841 raft_consensus.cc:2258] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.480921 5841 ts_tablet_manager.cc:1875] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:12.482031 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5f0b399a2d074f4ca3626ff17bd5518e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f0b399a2d074f4ca3626ff17bd5518e already in progress: creating tablet
I20240507 08:12:12.479874 5443 raft_consensus.cc:2454] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.483173 5656 leader_election.cc:302] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.483897 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1ddacc4556034a0892446b12705d92cd" } new_table_name: "default.Table_27ec04cf1e6643b496d0d1a979e3f4f9" modify_external_catalogs: true
I20240507 08:12:12.485291 7100 raft_consensus.cc:2781] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:12.485384 7100 raft_consensus.cc:482] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:12.485446 7100 raft_consensus.cc:3037] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.485750 5840 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.485859 5840 ts_tablet_manager.cc:1367] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.001s
I20240507 08:12:12.486505 5840 raft_consensus.cc:348] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.486675 5840 raft_consensus.cc:374] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.486716 5840 raft_consensus.cc:729] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.488101 7100 raft_consensus.cc:504] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.489254 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:12.489421 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_ec36ee6c38ab4cd29001e2a9488665c8 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677] alter complete (version 4)
I20240507 08:12:12.489657 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:12.490221 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b5e3d23bbb6439a87b303dff1b44a1d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:12.490434 5654 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.490998 5840 consensus_queue.cc:260] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.491214 7100 leader_election.cc:288] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.494203 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4b785d2845eb44679d48de42c13e1bdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:12.494710 5840 ts_tablet_manager.cc:1398] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
W20240507 08:12:12.494936 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b785d2845eb44679d48de42c13e1bdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.494975 5840 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:12.502914 5656 consensus_peers.cc:485] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.502926 7100 raft_consensus.cc:482] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.503134 7100 raft_consensus.cc:504] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.503317 5839 tablet_bootstrap.cc:492] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.503443 5839 ts_tablet_manager.cc:1367] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240507 08:12:12.503765 7100 leader_election.cc:288] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.504233 5839 raft_consensus.cc:348] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.504400 5839 raft_consensus.cc:374] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.504503 5839 raft_consensus.cc:729] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.504544 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.505023 5654 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.505851 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.496539 5840 tablet_bootstrap.cc:654] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.507956 5262 catalog_manager.cc:5484] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:12.508352 5262 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]). Expected version 2 got 0
I20240507 08:12:12.512777 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c373339c2d25449e8c18ce24b35e434e" } new_table_name: "default.Table_88cfc2e67b744004be89fcaf035f7894" modify_external_catalogs: true
I20240507 08:12:12.518002 5839 consensus_queue.cc:260] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.519306 7079 ts_tablet_manager.cc:1888] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:12.519421 7079 log.cc:1196] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/0118ef28b24641d68cf24ce9df966243
I20240507 08:12:12.519728 5839 ts_tablet_manager.cc:1398] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240507 08:12:12.520303 5839 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.522317 6513 ts_tablet_manager.cc:1888] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.522507 6513 log.cc:1196] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:12:12.522859 5584 raft_consensus.cc:3037] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.522929 6513 ts_tablet_manager.cc:1909] T 7841f5ebf5ed4f6eb28dcbd4087f1713 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:12.524292 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 (DEFAULT_TABLE table=default.Table_2a9bf32660374d72a70a5c5b228881a1 [id=ec337a65c5164f4f86e1718616de7edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.524457 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 (DEFAULT_TABLE table=default.Table_2a9bf32660374d72a70a5c5b228881a1 [id=ec337a65c5164f4f86e1718616de7edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.524663 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a14cf04d7a4ce3a2df939d0db515b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.524797 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a14cf04d7a4ce3a2df939d0db515b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.524919 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 918) took 3902 ms. Trace:
I20240507 08:12:12.525000 6513 rpcz_store.cc:276] 0507 08:12:08.622528 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955800 (+333272us) service_pool.cc:225] Handling call
0507 08:12:12.421643 (+3465843us) ts_tablet_manager.cc:1120] Deleting tablet 7841f5ebf5ed4f6eb28dcbd4087f1713
0507 08:12:12.421655 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.461699 (+ 40044us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.522218 (+ 60519us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.524898 (+ 2680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465741}
I20240507 08:12:12.526512 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 (table default.Table_6fd5e754c95b480a8ecd9e7f593b7260 [id=0b044cc019d845b9a033666c78e88492]) successfully deleted
I20240507 08:12:12.526909 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 919) took 3874 ms. Trace:
I20240507 08:12:12.526983 6513 rpcz_store.cc:276] 0507 08:12:08.652091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.955896 (+303805us) service_pool.cc:225] Handling call
0507 08:12:12.526789 (+3570893us) ts_tablet_manager.cc:1120] Deleting tablet d2240fd14155495a8c97a0af96090faf
0507 08:12:12.526800 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.526894 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3570790}
W20240507 08:12:12.527405 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d2240fd14155495a8c97a0af96090faf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2240fd14155495a8c97a0af96090faf already in progress: creating tablet
I20240507 08:12:12.527895 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 920) took 3864 ms. Trace:
I20240507 08:12:12.527961 6513 rpcz_store.cc:276] 0507 08:12:08.663518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.956007 (+292489us) service_pool.cc:225] Handling call
0507 08:12:12.527782 (+3571775us) ts_tablet_manager.cc:1120] Deleting tablet 8fc65070bbae4db19c5d14b6670d599c
0507 08:12:12.527794 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.527878 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3571683}
W20240507 08:12:12.528313 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 8fc65070bbae4db19c5d14b6670d599c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc65070bbae4db19c5d14b6670d599c already in progress: creating tablet
I20240507 08:12:12.529007 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 921) took 3865 ms. Trace:
I20240507 08:12:12.529071 6513 rpcz_store.cc:276] 0507 08:12:08.663641 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.956105 (+292464us) service_pool.cc:225] Handling call
0507 08:12:12.528915 (+3572810us) ts_tablet_manager.cc:1120] Deleting tablet ec3e1823f9ff4f19be2b13cdcd706f59
0507 08:12:12.528923 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.528993 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3572710}
W20240507 08:12:12.530874 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ec3e1823f9ff4f19be2b13cdcd706f59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec3e1823f9ff4f19be2b13cdcd706f59 already in progress: creating tablet
I20240507 08:12:12.531658 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 922) took 3857 ms. Trace:
I20240507 08:12:12.532013 6513 rpcz_store.cc:276] 0507 08:12:08.674069 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.956218 (+282149us) service_pool.cc:225] Handling call
0507 08:12:12.531563 (+3575345us) ts_tablet_manager.cc:1120] Deleting tablet 5b0c793dbc884efd92b49ae7798206a2
0507 08:12:12.531571 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.531644 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3575237}
W20240507 08:12:12.532624 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5b0c793dbc884efd92b49ae7798206a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0c793dbc884efd92b49ae7798206a2 already in progress: creating tablet
I20240507 08:12:12.533035 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 931) took 3567 ms. Trace:
I20240507 08:12:12.533600 6513 rpcz_store.cc:276] 0507 08:12:08.965482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.068806 (+103324us) service_pool.cc:225] Handling call
0507 08:12:12.532938 (+3464132us) ts_tablet_manager.cc:1120] Deleting tablet e7eef600626a4961ab1fae27162537bb
0507 08:12:12.532947 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.533020 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3394143}
W20240507 08:12:12.534055 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e7eef600626a4961ab1fae27162537bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7eef600626a4961ab1fae27162537bb already in progress: creating tablet
I20240507 08:12:12.534868 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 934) took 3519 ms. Trace:
I20240507 08:12:12.536017 6513 rpcz_store.cc:276] 0507 08:12:09.014958 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.138869 (+123911us) service_pool.cc:225] Handling call
0507 08:12:12.534772 (+3395903us) ts_tablet_manager.cc:1120] Deleting tablet d94eb57a25bc440196abe6861ca19b95
0507 08:12:12.534782 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.534853 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3395536}
W20240507 08:12:12.536485 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d94eb57a25bc440196abe6861ca19b95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d94eb57a25bc440196abe6861ca19b95 already in progress: creating tablet
I20240507 08:12:12.535930 5840 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.537400 5840 ts_tablet_manager.cc:1367] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240507 08:12:12.538341 5840 raft_consensus.cc:348] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.538604 5840 raft_consensus.cc:374] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.538720 5840 raft_consensus.cc:729] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.539002 5840 consensus_queue.cc:260] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:12.540290 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 56771302b1654642b9eaa21e459065ee has no consensus state
I20240507 08:12:12.542179 5840 ts_tablet_manager.cc:1398] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:12.542549 5840 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:12.544173 5840 tablet_bootstrap.cc:654] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.519744 7079 ts_tablet_manager.cc:1909] T 0118ef28b24641d68cf24ce9df966243 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:12.545260 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 0118ef28b24641d68cf24ce9df966243 (table default.Table_727fd7e435fd40368f63b993db76af91 [id=786a00558b0f42cd9700cd3200dfe1ac]) successfully deleted
I20240507 08:12:12.547274 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 (DEFAULT_TABLE table=default.Table_2a9bf32660374d72a70a5c5b228881a1 [id=ec337a65c5164f4f86e1718616de7edb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.547859 5445 raft_consensus.cc:2454] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.548180 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04a14cf04d7a4ce3a2df939d0db515b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.548842 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b27cc38d0c4c4083900cd6765a5cf6e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:12.549031 5443 raft_consensus.cc:3037] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:12.553661 7278 log.cc:924] Time spent T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20240507 08:12:12.555410 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 630e7da111304190979a9dbff263e982 (DEFAULT_TABLE table=default.Table_46792e71ef0e46df88355bf7c54afbd1 [id=08efbbb5b16447c5ba97dff01650fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.557439 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 630e7da111304190979a9dbff263e982 (DEFAULT_TABLE table=default.Table_46792e71ef0e46df88355bf7c54afbd1 [id=08efbbb5b16447c5ba97dff01650fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.557649 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 11290d416c9b476f8f7fdb22e424df01 (DEFAULT_TABLE table=default.Table_ad968e67c91142a99564460bd9f033d4 [id=51e543e15c1d4b65b7478a7082063672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.565423 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc280b83aa64f32bb12c2cd49f7799f (DEFAULT_TABLE table=default.Table_a51f8aaf8b364d69b4f79ed4cceb5157 [id=edc015aef6304afa99a7c3df52dd6cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.566157 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2b14c80b96bb43088e1d1354a1dbdd31
I20240507 08:12:12.566459 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1578a63b4e3e4743b1057fa69653cbef
I20240507 08:12:12.568503 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet 1578a63b4e3e4743b1057fa69653cbef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.568807 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.570238 7079 tablet_replica.cc:330] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:12.544611 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 932) took 3578 ms. Trace:
I20240507 08:12:12.548962 5656 leader_election.cc:302] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.558574 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 630e7da111304190979a9dbff263e982 (DEFAULT_TABLE table=default.Table_46792e71ef0e46df88355bf7c54afbd1 [id=08efbbb5b16447c5ba97dff01650fce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.560819 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 11290d416c9b476f8f7fdb22e424df01 (DEFAULT_TABLE table=default.Table_ad968e67c91142a99564460bd9f033d4 [id=51e543e15c1d4b65b7478a7082063672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.571522 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11290d416c9b476f8f7fdb22e424df01. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.572077 5656 consensus_peers.cc:485] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7ba2e8d1e01047fd9cc1c192e9628750. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.572347 6784 raft_consensus.cc:2781] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:12.572417 6784 raft_consensus.cc:482] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:12.572479 6784 raft_consensus.cc:3037] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.572764 7100 consensus_queue.cc:1035] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.568360 5696 tablet_service.cc:1514] Processing DeleteTablet for tablet 1578a63b4e3e4743b1057fa69653cbef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.573421 6513 rpcz_store.cc:276] 0507 08:12:08.965683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.070315 (+104632us) service_pool.cc:225] Handling call
0507 08:12:12.544490 (+3474175us) ts_tablet_manager.cc:1120] Deleting tablet bfa2139385d34c37a7ac14d78480a779
0507 08:12:12.544499 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.544593 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3404587}
I20240507 08:12:12.573992 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 933) took 3604 ms. Trace:
I20240507 08:12:12.574126 6513 rpcz_store.cc:276] 0507 08:12:08.969495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.070593 (+101098us) service_pool.cc:225] Handling call
0507 08:12:12.573883 (+3503290us) ts_tablet_manager.cc:1120] Deleting tablet f2de26190bc44916809856d84ad68345
0507 08:12:12.573893 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.573976 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3433813}
I20240507 08:12:12.574477 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 948) took 2483 ms. Trace:
I20240507 08:12:12.574733 7079 raft_consensus.cc:2229] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.575075 6784 raft_consensus.cc:504] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:12.575721 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bfa2139385d34c37a7ac14d78480a779 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa2139385d34c37a7ac14d78480a779 already in progress: creating tablet
W20240507 08:12:12.575960 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f2de26190bc44916809856d84ad68345 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2de26190bc44916809856d84ad68345 already in progress: creating tablet
I20240507 08:12:12.568665 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.576135 6784 leader_election.cc:288] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.576445 6513 rpcz_store.cc:276] 0507 08:12:10.091489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.096291 (+ 4802us) service_pool.cc:225] Handling call
0507 08:12:12.574386 (+2478095us) ts_tablet_manager.cc:1120] Deleting tablet 1ab94b1a09dd4969b928b017229ce4bf
0507 08:12:12.574396 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.574464 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2477878}
I20240507 08:12:12.576776 7079 raft_consensus.cc:2258] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.577102 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 947) took 2485 ms. Trace:
W20240507 08:12:12.577207 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1ab94b1a09dd4969b928b017229ce4bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab94b1a09dd4969b928b017229ce4bf already in progress: creating tablet
I20240507 08:12:12.576788 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:12.577778 7079 ts_tablet_manager.cc:1875] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:12.577809 5654 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.578315 6513 rpcz_store.cc:276] 0507 08:12:10.091374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.108971 (+ 17597us) service_pool.cc:225] Handling call
0507 08:12:12.577001 (+2468030us) ts_tablet_manager.cc:1120] Deleting tablet 06898414c5494cd380e0c932f4589014
0507 08:12:12.577013 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.577089 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467423}
I20240507 08:12:12.578758 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630e7da111304190979a9dbff263e982. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.578785 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 06898414c5494cd380e0c932f4589014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06898414c5494cd380e0c932f4589014 already in progress: creating tablet
I20240507 08:12:12.580597 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 949) took 2486 ms. Trace:
I20240507 08:12:12.580693 6513 rpcz_store.cc:276] 0507 08:12:10.093828 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.110214 (+ 16386us) service_pool.cc:225] Handling call
0507 08:12:12.580465 (+2470251us) ts_tablet_manager.cc:1120] Deleting tablet 2e44c3f6f96e417b92878d8ba4ff88a7
0507 08:12:12.580478 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.580579 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2470063}
W20240507 08:12:12.581151 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e44c3f6f96e417b92878d8ba4ff88a7 already in progress: creating tablet
I20240507 08:12:12.581564 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 951) took 2458 ms. Trace:
I20240507 08:12:12.581660 6513 rpcz_store.cc:276] 0507 08:12:10.123220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.123863 (+ 643us) service_pool.cc:225] Handling call
0507 08:12:12.581440 (+2457577us) ts_tablet_manager.cc:1120] Deleting tablet d17b6a2f4aa14f2a927fa9ad56a62a5a
0507 08:12:12.581451 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.581547 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2457391}
W20240507 08:12:12.582091 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17b6a2f4aa14f2a927fa9ad56a62a5a already in progress: creating tablet
I20240507 08:12:12.582666 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 950) took 2459 ms. Trace:
I20240507 08:12:12.582736 6513 rpcz_store.cc:276] 0507 08:12:10.123081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.124075 (+ 994us) service_pool.cc:225] Handling call
0507 08:12:12.582532 (+2458457us) ts_tablet_manager.cc:1120] Deleting tablet bd6ca1a20df748e097de097d1c2d7be3
0507 08:12:12.582543 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.582651 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458295}
W20240507 08:12:12.583199 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet bd6ca1a20df748e097de097d1c2d7be3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6ca1a20df748e097de097d1c2d7be3 already in progress: creating tablet
I20240507 08:12:12.583539 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 954) took 2428 ms. Trace:
I20240507 08:12:12.583628 6513 rpcz_store.cc:276] 0507 08:12:10.154834 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.161026 (+ 6192us) service_pool.cc:225] Handling call
0507 08:12:12.583455 (+2422429us) ts_tablet_manager.cc:1120] Deleting tablet c48da19c9b9e4207b267137d240adc5b
0507 08:12:12.583463 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.583524 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422256}
W20240507 08:12:12.584216 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c48da19c9b9e4207b267137d240adc5b because the tablet was not found. No further retry: Not found: Tablet not found: c48da19c9b9e4207b267137d240adc5b
I20240507 08:12:12.584612 6513 tablet_replica.cc:330] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:12.584846 6513 raft_consensus.cc:2229] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:12.585266 6513 raft_consensus.cc:2258] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.586141 6784 raft_consensus.cc:482] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.586277 6784 raft_consensus.cc:504] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.586937 6784 leader_election.cc:288] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:12.589504 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 1578a63b4e3e4743b1057fa69653cbef with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.589680 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b14c80b96bb43088e1d1354a1dbdd31 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.589807 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630e7da111304190979a9dbff263e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.590090 7101 consensus_queue.cc:1035] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.590644 7101 consensus_queue.cc:1035] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:12.591256 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa094773f3bf418492bcd5b5021fd429" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:12.591966 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.585979 6513 ts_tablet_manager.cc:1875] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:12.592669 5654 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.596416 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 11290d416c9b476f8f7fdb22e424df01 (DEFAULT_TABLE table=default.Table_ad968e67c91142a99564460bd9f033d4 [id=51e543e15c1d4b65b7478a7082063672]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.588589 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.598448 6880 raft_consensus.cc:482] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.598584 5447 raft_consensus.cc:3037] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.599334 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc280b83aa64f32bb12c2cd49f7799f (DEFAULT_TABLE table=default.Table_a51f8aaf8b364d69b4f79ed4cceb5157 [id=edc015aef6304afa99a7c3df52dd6cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.600203 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc280b83aa64f32bb12c2cd49f7799f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.603238 5443 raft_consensus.cc:2454] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:12.606878 5445 raft_consensus.cc:2454] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.607282 5656 leader_election.cc:302] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.607359 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11290d416c9b476f8f7fdb22e424df01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.608209 6784 raft_consensus.cc:2781] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:12.608302 6784 raft_consensus.cc:482] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:12.608371 6784 raft_consensus.cc:3037] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.610924 6784 raft_consensus.cc:504] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.611626 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b (DEFAULT_TABLE table=default.Table_695a3d2597174aa7bc8a61e91f85f9a7 [id=9ece12dbeb47441195f06688ef5b7140]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.615216 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b (DEFAULT_TABLE table=default.Table_695a3d2597174aa7bc8a61e91f85f9a7 [id=9ece12dbeb47441195f06688ef5b7140]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.615581 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.618034 7107 raft_consensus.cc:482] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.620163 5839 tablet_bootstrap.cc:654] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.623278 5555 tablet_service.cc:1467] Processing CreateTablet for tablet a24813c72abd41c3830c7392c6149151 (DEFAULT_TABLE table=default.Table_d75fc8323e274f5bb918bbcc918a336e [id=8f257099eafd4d0597cf16d0e35308fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.626632 6880 raft_consensus.cc:504] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.627813 5656 leader_election.cc:302] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.628275 6675 raft_consensus.cc:2781] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:12.628399 6675 raft_consensus.cc:686] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:12.628672 6675 consensus_queue.cc:237] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.629139 5690 tablet_service.cc:1467] Processing CreateTablet for tablet a24813c72abd41c3830c7392c6149151 (DEFAULT_TABLE table=default.Table_d75fc8323e274f5bb918bbcc918a336e [id=8f257099eafd4d0597cf16d0e35308fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.629755 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24813c72abd41c3830c7392c6149151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.636974 7101 raft_consensus.cc:482] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.637104 7101 raft_consensus.cc:504] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.637738 7101 leader_election.cc:288] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.638120 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.638756 5654 leader_election.cc:341] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.640388 6784 leader_election.cc:288] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:12.640947 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 2c92727908174c4d9f6aa659569027ff (DEFAULT_TABLE table=default.Table_446a41da25a64da8969bccf51fab7fa5 [id=6e246804369a45a2a4d5400a2ebbfec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.641726 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:12.642364 5654 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:12.643564 5654 consensus_peers.cc:485] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d6f8f4205f1f48abaf4fcd7113ad44a8. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.643750 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 630e7da111304190979a9dbff263e982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.643926 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.648566 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 2572985bc2444dc7b051f8fe008e3ad4 (table default.Table_47188cab4dee4904b2a5099a06b2a652 [id=3b28b0b34e044bdfbe5bc7b6743d081e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2572985bc2444dc7b051f8fe008e3ad4 consensus state has no leader
I20240507 08:12:12.650053 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c92727908174c4d9f6aa659569027ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.654356 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24813c72abd41c3830c7392c6149151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.657933 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11290d416c9b476f8f7fdb22e424df01. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.659698 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5855bc973d354a69a0579d9a76b3f92b
I20240507 08:12:12.660356 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet e2b27de4d2d84a2f8c177a231c985785 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:12.661586 5547 tablet_service.cc:1514] Processing DeleteTablet for tablet 5855bc973d354a69a0579d9a76b3f92b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.663170 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc280b83aa64f32bb12c2cd49f7799f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.663130 5264 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:12.664084 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb97573b1b0e4931be83172a373e0592"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.664793 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb97573b1b0e4931be83172a373e0592 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.666769 5840 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.666934 5840 ts_tablet_manager.cc:1367] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.124s user 0.005s sys 0.000s
I20240507 08:12:12.667686 5840 raft_consensus.cc:348] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.668174 5840 raft_consensus.cc:374] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.668273 5840 raft_consensus.cc:729] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.668565 7214 raft_consensus.cc:482] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.668520 5840 consensus_queue.cc:260] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.668697 7214 raft_consensus.cc:504] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.669131 5840 ts_tablet_manager.cc:1398] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:12.669370 7214 leader_election.cc:288] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.669608 5840 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:12.670969 5520 leader_election.cc:341] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 4de7053cfaea4a54a402c9d826613c05
I20240507 08:12:12.671197 5840 tablet_bootstrap.cc:654] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:12.671835 5517 leader_election.cc:341] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 4de7053cfaea4a54a402c9d826613c05
I20240507 08:12:12.671934 5517 leader_election.cc:302] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:12.672649 7214 raft_consensus.cc:2726] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:12.673266 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_599be0c8696c4c4f94c232b1668f717c" } new_table_name: "default.Table_d036e0ee03b344c1872fd1624fe92142" modify_external_catalogs: true
I20240507 08:12:12.675665 7079 ts_tablet_manager.cc:1888] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.675976 7079 log.cc:1196] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/dc74342c8aee4017a212aa88a7f3dc06
I20240507 08:12:12.676862 7079 ts_tablet_manager.cc:1909] T dc74342c8aee4017a212aa88a7f3dc06 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:12.677668 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet dc74342c8aee4017a212aa88a7f3dc06 (table default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]) successfully deleted
I20240507 08:12:12.678041 7079 tablet_replica.cc:330] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:12.678282 7079 raft_consensus.cc:2229] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:12.680058 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]). Expected version 2 got 0
I20240507 08:12:12.681887 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4c3768696f85440290e9da3a265d45d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.682567 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c3768696f85440290e9da3a265d45d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.684628 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_97fdb4f0eb6f4975882ec9a27516eb43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.685400 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97fdb4f0eb6f4975882ec9a27516eb43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.687633 5261 catalog_manager.cc:5484] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:12.688295 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]). Expected version 2 got 0
I20240507 08:12:12.681509 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_59179182d23d4f1f9e0311758a287f76" } new_table_name: "default.Table_9baae6b9dc5848a884fa67e9130222c3" modify_external_catalogs: true
I20240507 08:12:12.695101 5447 raft_consensus.cc:2454] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:12.695384 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 5855bc973d354a69a0579d9a76b3f92b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.695533 5656 leader_election.cc:302] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:12.696058 7100 raft_consensus.cc:2781] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:12.696174 7100 raft_consensus.cc:686] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:12.696398 7100 consensus_queue.cc:237] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.699417 5696 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee77559e2044c50863c43b397257267 (DEFAULT_TABLE table=default.Table_e9226c81998d47d7a3359930560d2953 [id=462b054d595e47aea861a5b0fcfb6af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.699831 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee77559e2044c50863c43b397257267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.708148 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 37a23d112fcd4ea1b0c2f86133e82e32
I20240507 08:12:12.708609 5255 catalog_manager.cc:6554] ReplaceTablet: tablet 37a23d112fcd4ea1b0c2f86133e82e32 deleted and replaced by tablet b65e251f3d5c4414afe413a54a519d85
I20240507 08:12:12.709035 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8b032f4a53674fa3a9bcb78cd817e5e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.709622 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b032f4a53674fa3a9bcb78cd817e5e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.710995 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a3535f1fbfe34f6e95e9895771ae1350
I20240507 08:12:12.711206 7107 raft_consensus.cc:504] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.711544 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet d2240fd14155495a8c97a0af96090faf from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:12.694316 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "957775c01be94cc081db344c95a81310" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:12.712069 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3f83db914b5b4f38a335b48699966012" } new_table_name: "default.Table_2eeab823921445deb9bd071d18014442" modify_external_catalogs: true
I20240507 08:12:12.712925 5686 tablet_service.cc:1514] Processing DeleteTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.717818 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069530908792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:12:12.718052 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:12.724316 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 2c92727908174c4d9f6aa659569027ff (DEFAULT_TABLE table=default.Table_446a41da25a64da8969bccf51fab7fa5 [id=6e246804369a45a2a4d5400a2ebbfec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.724795 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c92727908174c4d9f6aa659569027ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.725638 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b (DEFAULT_TABLE table=default.Table_695a3d2597174aa7bc8a61e91f85f9a7 [id=9ece12dbeb47441195f06688ef5b7140]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.726150 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a24813c72abd41c3830c7392c6149151 (DEFAULT_TABLE table=default.Table_d75fc8323e274f5bb918bbcc918a336e [id=8f257099eafd4d0597cf16d0e35308fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.725478 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 5bc280b83aa64f32bb12c2cd49f7799f (DEFAULT_TABLE table=default.Table_a51f8aaf8b364d69b4f79ed4cceb5157 [id=edc015aef6304afa99a7c3df52dd6cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.726809 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:12.731374 7101 raft_consensus.cc:482] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:12.731626 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf0dc818b10a4e1e921452a83a5c05ee consensus state has no leader
I20240507 08:12:12.731520 7101 raft_consensus.cc:504] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.732115 7101 leader_election.cc:288] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.732312 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c1932ffe7d37455c8aa6bb83ffd11498
W20240507 08:12:12.733448 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c053869c840e4773a3c8e07a8b426b28 consensus state has no leader
I20240507 08:12:12.747001 5686 tablet_service.cc:1467] Processing CreateTablet for tablet daa3ee2274004cf09e21aa6861049df7 (DEFAULT_TABLE table=default.Table_ebd8a847575e430eb26365ca4cf1928a [id=e8a0168537084869a0ec8800e9653e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.747153 5696 tablet_service.cc:1467] Processing CreateTablet for tablet d9bae1b289994a2c8ea89be89ead6a7b (DEFAULT_TABLE table=default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb [id=29657ce4dafa47ce8bfcc8268c4aad3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.747480 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa3ee2274004cf09e21aa6861049df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.747694 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bae1b289994a2c8ea89be89ead6a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.755599 5686 tablet_service.cc:1467] Processing CreateTablet for tablet f975fac7312c4de39fbeada171119ed7 (DEFAULT_TABLE table=default.Table_6571d5b9dda642549111e4a278c03f33 [id=b7ad67f7b9c64478b1439eba36a11b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.756031 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f975fac7312c4de39fbeada171119ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.757843 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 (DEFAULT_TABLE table=default.Table_908d49dcd5814349aee9767384232deb [id=51822bd6c3b0427789251307869f9109]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.758215 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ffac4b4f34551ba930f0a259e78a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.762054 5696 tablet_service.cc:1467] Processing CreateTablet for tablet a1f9ee9f367e42308b3374509bdd6c9f (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.762434 5696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f9ee9f367e42308b3374509bdd6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.766410 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 8249d1740fe44fa48b30d4216dafdc8b (DEFAULT_TABLE table=default.Table_ff3b0a8492a246b3be16662405bc7a09 [id=6a249d0fe0664a13ab3e20fee6a5e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.772294 5689 tablet_service.cc:1467] Processing CreateTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 (DEFAULT_TABLE table=default.Table_35152415e8264b5495c852cb2a68506c [id=11c22b88f23d455aabba69eca7ac2f30]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:12.774039 5654 consensus_peers.cc:485] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.775797 5693 tablet_service.cc:1467] Processing CreateTablet for tablet d3c4e1627f274501b7cb98a51a5ec76b (DEFAULT_TABLE table=default.Table_5392502ff2394c5d947c372642154554 [id=ab30b27c34eb4b459cf28e16755cb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.789330 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 with delete_type TABLET_DATA_DELETED (replaced by tablet b65e251f3d5c4414afe413a54a519d85) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.789633 6784 raft_consensus.cc:482] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.789762 6784 raft_consensus.cc:504] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.791163 6784 leader_election.cc:288] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:12.793064 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 5855bc973d354a69a0579d9a76b3f92b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:11 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.794946 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 15563bbb49084b58ac28b3f0cfab37a9 (DEFAULT_TABLE table=default.Table_2c3c93e036544172b7ddb34378d99f86 [id=6a5db850819343a687ac2a8d31e4c7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.800410 6880 leader_election.cc:288] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.801299 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe932fe185466bbd602aea7ef70250 (DEFAULT_TABLE table=default.Table_bb71acdadad640e7a4b25604ae4f80fb [id=e98c887398a349d890d90e50ce58e1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.802356 5445 raft_consensus.cc:3037] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:12.803550 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 701d80318e4d42ecb6e3cb5d5e7fe92b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.804767 5443 raft_consensus.cc:2454] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:12.805437 5692 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cb7339e6f443eb22a458c6ee9504d (DEFAULT_TABLE table=default.Table_27e9e4ffb901470db2653f82d93365ba [id=ae6f969dfaed4332ae45ba6bb8fe1615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.807518 5684 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a283ca27340cf88cb32832c41d298 (DEFAULT_TABLE table=default.Table_92a6805df9e7430dbaff524aaafe783a [id=bab6fae082b445498fb03f446e62f691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.807654 7107 leader_election.cc:288] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.809502 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 219bad2b972c45a8930d6f47ca361858 (DEFAULT_TABLE table=default.Table_089a4f3cc7ad43499020b5c9d1e6f969 [id=cce060a4ed054828b4f6ba828f717f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.811759 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 (DEFAULT_TABLE table=default.Table_6da541994d9c4b42b19de81f931ae930 [id=2c732a9c1372437caa4db6dc50707970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.814260 5699 tablet_service.cc:1467] Processing CreateTablet for tablet a769e920141f4ba288ebf85e3a18e791 (DEFAULT_TABLE table=default.Table_faf3a930a35b4a80aa6be69297d5ed7b [id=ff820582a283406db1942344c0ae6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.814520 6675 raft_consensus.cc:482] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.814626 6675 raft_consensus.cc:504] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:12.815691 6675 leader_election.cc:288] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:12.816231 6880 raft_consensus.cc:482] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.817286 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a24813c72abd41c3830c7392c6149151. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.701203 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee77559e2044c50863c43b397257267 (DEFAULT_TABLE table=default.Table_e9226c81998d47d7a3359930560d2953 [id=462b054d595e47aea861a5b0fcfb6af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.708408 7214 raft_consensus.cc:482] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.817943 7214 raft_consensus.cc:504] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.818259 7107 raft_consensus.cc:482] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.818380 7107 raft_consensus.cc:504] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.818646 7214 leader_election.cc:288] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.717128 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 with delete_type TABLET_DATA_DELETED (replaced by tablet b65e251f3d5c4414afe413a54a519d85) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.720678 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.721287 5551 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a59c7257e74b4abaa4ea7a1ee9136d with delete_type TABLET_DATA_DELETED (replaced by tablet 120bce4b20714fc1a0e1237abc00d5bb) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.733386 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.818960 7107 leader_election.cc:288] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:12.739078 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.741911 5547 tablet_service.cc:1467] Processing CreateTablet for tablet daa3ee2274004cf09e21aa6861049df7 (DEFAULT_TABLE table=default.Table_ebd8a847575e430eb26365ca4cf1928a [id=e8a0168537084869a0ec8800e9653e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.819114 6880 raft_consensus.cc:504] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.819409 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa3ee2274004cf09e21aa6861049df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.819747 6880 leader_election.cc:288] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:12.819900 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee77559e2044c50863c43b397257267. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.820587 5517 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:12:12.747175 5546 tablet_service.cc:1467] Processing CreateTablet for tablet f975fac7312c4de39fbeada171119ed7 (DEFAULT_TABLE table=default.Table_6571d5b9dda642549111e4a278c03f33 [id=b7ad67f7b9c64478b1439eba36a11b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.821866 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f975fac7312c4de39fbeada171119ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.756409 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 (DEFAULT_TABLE table=default.Table_908d49dcd5814349aee9767384232deb [id=51822bd6c3b0427789251307869f9109]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.761405 5554 tablet_service.cc:1467] Processing CreateTablet for tablet a1f9ee9f367e42308b3374509bdd6c9f (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.825949 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ffac4b4f34551ba930f0a259e78a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.825944 5445 raft_consensus.cc:2454] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:12.826119 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f9ee9f367e42308b3374509bdd6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.761770 5548 tablet_service.cc:1467] Processing CreateTablet for tablet d9bae1b289994a2c8ea89be89ead6a7b (DEFAULT_TABLE table=default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb [id=29657ce4dafa47ce8bfcc8268c4aad3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.765019 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 8249d1740fe44fa48b30d4216dafdc8b (DEFAULT_TABLE table=default.Table_ff3b0a8492a246b3be16662405bc7a09 [id=6a249d0fe0664a13ab3e20fee6a5e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.827095 7079 raft_consensus.cc:2258] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:12.827344 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8249d1740fe44fa48b30d4216dafdc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.827659 7079 ts_tablet_manager.cc:1875] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:12.767345 5549 tablet_service.cc:1467] Processing CreateTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 (DEFAULT_TABLE table=default.Table_35152415e8264b5495c852cb2a68506c [id=11c22b88f23d455aabba69eca7ac2f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.829818 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc798eb4e02c456880ef3014ebbb7e24. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.830507 5654 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.774662 5563 tablet_service.cc:1467] Processing CreateTablet for tablet d3c4e1627f274501b7cb98a51a5ec76b (DEFAULT_TABLE table=default.Table_5392502ff2394c5d947c372642154554 [id=ab30b27c34eb4b459cf28e16755cb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.778056 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 15563bbb49084b58ac28b3f0cfab37a9 (DEFAULT_TABLE table=default.Table_2c3c93e036544172b7ddb34378d99f86 [id=6a5db850819343a687ac2a8d31e4c7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.789968 5839 tablet_bootstrap.cc:492] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.791005 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:12.833890 5839 ts_tablet_manager.cc:1367] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.314s user 0.020s sys 0.065s
I20240507 08:12:12.833917 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c4e1627f274501b7cb98a51a5ec76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.834496 5839 raft_consensus.cc:348] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.834671 5839 raft_consensus.cc:374] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.834715 5839 raft_consensus.cc:729] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.834884 5839 consensus_queue.cc:260] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:12.835273 5654 leader_election.cc:341] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.835330 5839 ts_tablet_manager.cc:1398] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240507 08:12:12.794377 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe932fe185466bbd602aea7ef70250 (DEFAULT_TABLE table=default.Table_bb71acdadad640e7a4b25604ae4f80fb [id=e98c887398a349d890d90e50ce58e1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.802080 5560 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cb7339e6f443eb22a458c6ee9504d (DEFAULT_TABLE table=default.Table_27e9e4ffb901470db2653f82d93365ba [id=ae6f969dfaed4332ae45ba6bb8fe1615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.806898 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a283ca27340cf88cb32832c41d298 (DEFAULT_TABLE table=default.Table_92a6805df9e7430dbaff524aaafe783a [id=bab6fae082b445498fb03f446e62f691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.808318 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 219bad2b972c45a8930d6f47ca361858 (DEFAULT_TABLE table=default.Table_089a4f3cc7ad43499020b5c9d1e6f969 [id=cce060a4ed054828b4f6ba828f717f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.810407 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 (DEFAULT_TABLE table=default.Table_6da541994d9c4b42b19de81f931ae930 [id=2c732a9c1372437caa4db6dc50707970]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.836239 5839 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.837714 5839 tablet_bootstrap.cc:654] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.838117 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bae1b289994a2c8ea89be89ead6a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.838639 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe932fe185466bbd602aea7ef70250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.839936 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.842144 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cb7339e6f443eb22a458c6ee9504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.811831 7300 raft_consensus.cc:482] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:12.812811 5556 tablet_service.cc:1467] Processing CreateTablet for tablet a769e920141f4ba288ebf85e3a18e791 (DEFAULT_TABLE table=default.Table_faf3a930a35b4a80aa6be69297d5ed7b [id=ff820582a283406db1942344c0ae6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.815788 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:12.843833 5654 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:12.843454 7300 raft_consensus.cc:504] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.845871 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a283ca27340cf88cb32832c41d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.845844 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:12.846717 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc798eb4e02c456880ef3014ebbb7e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.847182 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c4e1627f274501b7cb98a51a5ec76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.847604 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cb7339e6f443eb22a458c6ee9504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.847950 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a283ca27340cf88cb32832c41d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.848320 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219bad2b972c45a8930d6f47ca361858. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.848683 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c9c649ec3a4a3db414b78ef9792423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.849036 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a769e920141f4ba288ebf85e3a18e791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.849547 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8249d1740fe44fa48b30d4216dafdc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.850526 5547 tablet_service.cc:1467] Processing CreateTablet for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b (DEFAULT_TABLE table=default.Table_1633148e0c444d8eaedb7e613aa6ed3b [id=18a188c9ece44072a92126f52dd855a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.852075 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe932fe185466bbd602aea7ef70250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.852113 7300 leader_election.cc:288] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:12.852924 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c9c649ec3a4a3db414b78ef9792423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.853014 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b (DEFAULT_TABLE table=default.Table_1633148e0c444d8eaedb7e613aa6ed3b [id=18a188c9ece44072a92126f52dd855a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.855216 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 0929c348711341d9b3529f22ff2b3df9 (DEFAULT_TABLE table=default.Table_63a2b5e8edd14d6aa283025fbf8df787 [id=a374bc2801e64de784cc8485357a8bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.855625 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929c348711341d9b3529f22ff2b3df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.855917 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 0929c348711341d9b3529f22ff2b3df9 (DEFAULT_TABLE table=default.Table_63a2b5e8edd14d6aa283025fbf8df787 [id=a374bc2801e64de784cc8485357a8bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.856541 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11c0fe13d34629b0491a2627f83048 (DEFAULT_TABLE table=default.Table_16de5f84b4f3402ebb29a677269ab0a0 [id=2101b18a04f74d6e8da8c7473b710775]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.857018 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11c0fe13d34629b0491a2627f83048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.858188 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15563bbb49084b58ac28b3f0cfab37a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.860812 5841 ts_tablet_manager.cc:1888] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:12.861014 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bc280b83aa64f32bb12c2cd49f7799f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.863003 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a769e920141f4ba288ebf85e3a18e791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.863615 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_47d427710f564bdebf6ad097dbfde129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.864197 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47d427710f564bdebf6ad097dbfde129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.865517 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a668c8aa1ac34ee799dac4150b2e59ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:12.865779 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219bad2b972c45a8930d6f47ca361858. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:12.865964 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a668c8aa1ac34ee799dac4150b2e59ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.867039 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_381633ec2ac14aa4a5325fd26dafde7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.867475 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_381633ec2ac14aa4a5325fd26dafde7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.868649 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_77bee6f5db85437bb6ecccd959ba929f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.869071 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77bee6f5db85437bb6ecccd959ba929f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.870175 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11c0fe13d34629b0491a2627f83048 (DEFAULT_TABLE table=default.Table_16de5f84b4f3402ebb29a677269ab0a0 [id=2101b18a04f74d6e8da8c7473b710775]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.872874 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.873009 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6a42d94f854c4b529b014e54387d2869"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:12.874008 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a42d94f854c4b529b014e54387d2869 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:12.874610 5700 tablet_service.cc:1467] Processing CreateTablet for tablet f748727d3a88442d90cf81bae48acd36 (DEFAULT_TABLE table=default.Table_9e717d2407ba4a54af3626573fb38c0b [id=489e7810cca5442485edf95b885b84cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.875042 5697 tablet_service.cc:1467] Processing CreateTablet for tablet d8787ec1aec54ccaacf9096905742756 (DEFAULT_TABLE table=default.Table_c75f62ac2a774ef79d3238ef5ff7d317 [id=9abb4ed23bd04ce8a0a3b4cb758c510d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.875394 5559 tablet_service.cc:1467] Processing CreateTablet for tablet f748727d3a88442d90cf81bae48acd36 (DEFAULT_TABLE table=default.Table_9e717d2407ba4a54af3626573fb38c0b [id=489e7810cca5442485edf95b885b84cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.874838 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d8787ec1aec54ccaacf9096905742756 (DEFAULT_TABLE table=default.Table_c75f62ac2a774ef79d3238ef5ff7d317 [id=9abb4ed23bd04ce8a0a3b4cb758c510d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.876617 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.880582 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748727d3a88442d90cf81bae48acd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.880832 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet a3535f1fbfe34f6e95e9895771ae1350 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.880905 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet c1932ffe7d37455c8aa6bb83ffd11498 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.881471 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 8ee77559e2044c50863c43b397257267 (DEFAULT_TABLE table=default.Table_e9226c81998d47d7a3359930560d2953 [id=462b054d595e47aea861a5b0fcfb6af0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.881671 5416 tablet_service.cc:1467] Processing CreateTablet for tablet f975fac7312c4de39fbeada171119ed7 (DEFAULT_TABLE table=default.Table_6571d5b9dda642549111e4a278c03f33 [id=b7ad67f7b9c64478b1439eba36a11b82]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.881911 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ee77559e2044c50863c43b397257267. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.881842 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 8249d1740fe44fa48b30d4216dafdc8b (DEFAULT_TABLE table=default.Table_ff3b0a8492a246b3be16662405bc7a09 [id=6a249d0fe0664a13ab3e20fee6a5e322]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.882172 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8249d1740fe44fa48b30d4216dafdc8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.882179 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 (DEFAULT_TABLE table=default.Table_908d49dcd5814349aee9767384232deb [id=51822bd6c3b0427789251307869f9109]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.882404 5412 tablet_service.cc:1467] Processing CreateTablet for tablet daa3ee2274004cf09e21aa6861049df7 (DEFAULT_TABLE table=default.Table_ebd8a847575e430eb26365ca4cf1928a [id=e8a0168537084869a0ec8800e9653e01]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.882560 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 669ffac4b4f34551ba930f0a259e78a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.882715 5415 tablet_service.cc:1467] Processing CreateTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 (DEFAULT_TABLE table=default.Table_35152415e8264b5495c852cb2a68506c [id=11c22b88f23d455aabba69eca7ac2f30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.882938 5425 tablet_service.cc:1467] Processing CreateTablet for tablet a1f9ee9f367e42308b3374509bdd6c9f (DEFAULT_TABLE table=default.Table_0ce315fc8bc94622b827617e4cd26c98 [id=6437f89f80974fe1b780ab7fdcfad354]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.883388 5421 tablet_service.cc:1467] Processing CreateTablet for tablet d9bae1b289994a2c8ea89be89ead6a7b (DEFAULT_TABLE table=default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb [id=29657ce4dafa47ce8bfcc8268c4aad3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.883450 5426 tablet_service.cc:1467] Processing CreateTablet for tablet d3c4e1627f274501b7cb98a51a5ec76b (DEFAULT_TABLE table=default.Table_5392502ff2394c5d947c372642154554 [id=ab30b27c34eb4b459cf28e16755cb567]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.882727 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet daa3ee2274004cf09e21aa6861049df7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.883865 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f975fac7312c4de39fbeada171119ed7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.884099 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1f9ee9f367e42308b3374509bdd6c9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.884110 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 15563bbb49084b58ac28b3f0cfab37a9 (DEFAULT_TABLE table=default.Table_2c3c93e036544172b7ddb34378d99f86 [id=6a5db850819343a687ac2a8d31e4c7b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.884856 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 51fe932fe185466bbd602aea7ef70250 (DEFAULT_TABLE table=default.Table_bb71acdadad640e7a4b25604ae4f80fb [id=e98c887398a349d890d90e50ce58e1a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.885344 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 2c92727908174c4d9f6aa659569027ff (DEFAULT_TABLE table=default.Table_446a41da25a64da8969bccf51fab7fa5 [id=6e246804369a45a2a4d5400a2ebbfec6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.885701 5409 tablet_service.cc:1467] Processing CreateTablet for tablet fb2cb7339e6f443eb22a458c6ee9504d (DEFAULT_TABLE table=default.Table_27e9e4ffb901470db2653f82d93365ba [id=ae6f969dfaed4332ae45ba6bb8fe1615]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.886327 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 7a6a283ca27340cf88cb32832c41d298 (DEFAULT_TABLE table=default.Table_92a6805df9e7430dbaff524aaafe783a [id=bab6fae082b445498fb03f446e62f691]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.886518 5841 log.cc:1196] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:12.887796 5841 ts_tablet_manager.cc:1909] T b0b9f85cd78748d3afce83b64b3f823b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:12.889705 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 871) took 5483 ms. Trace:
I20240507 08:12:12.890064 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2759ebfa0840f8b21d749354768c8d (DEFAULT_TABLE table=default.Table_fb6e602d297c493fa3c7969a29ea399b [id=5101edaa20304b709afaa20b4d04fc84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.889984 5841 rpcz_store.cc:276] 0507 08:12:07.406145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.406419 (+ 274us) service_pool.cc:225] Handling call
0507 08:12:12.474630 (+5068211us) ts_tablet_manager.cc:1120] Deleting tablet b0b9f85cd78748d3afce83b64b3f823b
0507 08:12:12.476013 (+ 1383us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.593078 (+117065us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.817009 (+223931us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:12.889621 (+ 72612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5068009}
I20240507 08:12:12.890583 5839 tablet_bootstrap.cc:492] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:12.890863 5839 ts_tablet_manager.cc:1367] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.003s
I20240507 08:12:12.891214 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 870) took 5494 ms. Trace:
I20240507 08:12:12.892278 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2759ebfa0840f8b21d749354768c8d (DEFAULT_TABLE table=default.Table_fb6e602d297c493fa3c7969a29ea399b [id=5101edaa20304b709afaa20b4d04fc84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.891521 5841 rpcz_store.cc:276] 0507 08:12:07.396086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.396168 (+ 82us) service_pool.cc:225] Handling call
0507 08:12:12.890742 (+5494574us) ts_tablet_manager.cc:1120] Deleting tablet 574a47894d274360891253534bf215a5
0507 08:12:12.890753 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.891029 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5483143}
I20240507 08:12:12.894349 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 1578a63b4e3e4743b1057fa69653cbef (DEFAULT_TABLE table=default.Table_d3257e1057da421d901ffdb965228250 [id=531bedba63c14afb9a6d0dec9e2049a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.897853 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 1578a63b4e3e4743b1057fa69653cbef (DEFAULT_TABLE table=default.Table_d3257e1057da421d901ffdb965228250 [id=531bedba63c14afb9a6d0dec9e2049a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:12.899580 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578a63b4e3e4743b1057fa69653cbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.900692 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15563bbb49084b58ac28b3f0cfab37a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.901743 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 37a23d112fcd4ea1b0c2f86133e82e32 with delete_type TABLET_DATA_DELETED (replaced by tablet b65e251f3d5c4414afe413a54a519d85) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:12.902761 5841 tablet_replica.cc:330] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:12.909858 5839 raft_consensus.cc:348] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.910065 5839 raft_consensus.cc:374] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.910115 5839 raft_consensus.cc:729] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.911486 5839 consensus_queue.cc:260] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.912261 5839 ts_tablet_manager.cc:1398] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
I20240507 08:12:12.918345 5839 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:12.920948 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15563bbb49084b58ac28b3f0cfab37a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.921782 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2759ebfa0840f8b21d749354768c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.925228 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8787ec1aec54ccaacf9096905742756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.925933 5839 tablet_bootstrap.cc:654] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:12.945832 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.946054 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.946192 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 with delete_type TABLET_DATA_DELETED (replaced by tablet 8d6b0ccf73cc46a3a7815e5d09ed7614) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:12.946322 5694 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:12.951009 5654 consensus_peers.cc:485] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:12.962877 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11c0fe13d34629b0491a2627f83048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.965801 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748727d3a88442d90cf81bae48acd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.968183 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8787ec1aec54ccaacf9096905742756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.970512 7079 ts_tablet_manager.cc:1888] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:12.972221 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2759ebfa0840f8b21d749354768c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.972579 5840 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:12.972800 5840 ts_tablet_manager.cc:1367] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.303s user 0.003s sys 0.002s
I20240507 08:12:12.974663 5840 raft_consensus.cc:348] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.975414 5840 raft_consensus.cc:374] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:12.975564 5840 raft_consensus.cc:729] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:12.976047 5840 consensus_queue.cc:260] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:12.977002 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578a63b4e3e4743b1057fa69653cbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:12.992163 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 835d7b779d724df4863bb13a5140dbf0 (table default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce]). Expected version 1 got 0
I20240507 08:12:12.995949 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6da541994d9c4b42b19de81f931ae930" } modify_external_catalogs: true
I20240507 08:12:12.996459 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e2faf177bdbe4c68b0a8d371c5a33fb2" } new_table_name: "default.Table_1509455bf80a45faa581f2b089fa0f22" modify_external_catalogs: true
I20240507 08:12:12.996546 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:12.997808 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6da541994d9c4b42b19de81f931ae930" } modify_external_catalogs: true
I20240507 08:12:13.000180 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929c348711341d9b3529f22ff2b3df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.000422 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.020859 6513 ts_tablet_manager.cc:1888] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.023981 5579 raft_consensus.cc:3037] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.037220 7079 log.cc:1196] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/ba7c1aaf55d74c979ec1fc0a57f15f44
I20240507 08:12:13.038369 7079 ts_tablet_manager.cc:1909] T ba7c1aaf55d74c979ec1fc0a57f15f44 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:13.039569 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (table default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]) successfully deleted
I20240507 08:12:13.040251 7079 tablet_replica.cc:330] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:13.040900 7079 raft_consensus.cc:2229] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.041024 7300 raft_consensus.cc:482] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.041388 7300 raft_consensus.cc:504] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.039988 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.042074 7300 leader_election.cc:288] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.042466 6513 log.cc:1196] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/7b4b84ab3f274185a2b838ebb969ab52
I20240507 08:12:13.042958 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44ec0526e1914221bc22b3e75ea01e0d" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.045400 5840 ts_tablet_manager.cc:1398] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.072s user 0.003s sys 0.000s
I20240507 08:12:13.046290 5840 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:13.051265 5654 consensus_peers.cc:485] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc74342c8aee4017a212aa88a7f3dc06. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.051441 6513 ts_tablet_manager.cc:1909] T 7b4b84ab3f274185a2b838ebb969ab52 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:13.063424 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 956) took 2908 ms. Trace:
I20240507 08:12:13.064411 7300 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.064584 7300 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.065163 7300 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.065747 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a6a283ca27340cf88cb32832c41d298. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.065968 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:13.067130 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55a535a1039d43b9b0bf2c61304c2d01 consensus state has no leader
I20240507 08:12:13.070003 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3c4e1627f274501b7cb98a51a5ec76b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.077260 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_398e8d420fb14944ac6abf845df55580" } modify_external_catalogs: true
I20240507 08:12:13.077469 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_398e8d420fb14944ac6abf845df55580" } modify_external_catalogs: true
I20240507 08:12:13.063763 6513 rpcz_store.cc:276] 0507 08:12:10.155064 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.159003 (+ 3939us) service_pool.cc:225] Handling call
0507 08:12:12.584561 (+2425558us) ts_tablet_manager.cc:1120] Deleting tablet 7b4b84ab3f274185a2b838ebb969ab52
0507 08:12:12.584572 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:12.686326 (+101754us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.020638 (+334312us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.063400 (+ 42762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404606}
W20240507 08:12:13.078110 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648113d9b8164cc59ba238ace8f65b39 consensus state has no leader
I20240507 08:12:13.078450 6513 tablet_replica.cc:330] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:13.078459 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b59f41c55cab461abd20c0fc9ae984bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.078953 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b59f41c55cab461abd20c0fc9ae984bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.078891 6513 raft_consensus.cc:2229] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:13.079675 6513 raft_consensus.cc:2258] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.080777 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 7b4b84ab3f274185a2b838ebb969ab52 (table default.Table_156b8cd7adba496a83c3269a663133f8 [id=36d7c838857e4be49bd3267d51890123]) successfully deleted
I20240507 08:12:13.080965 6513 ts_tablet_manager.cc:1875] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.086134 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 219bad2b972c45a8930d6f47ca361858 (DEFAULT_TABLE table=default.Table_089a4f3cc7ad43499020b5c9d1e6f969 [id=cce060a4ed054828b4f6ba828f717f2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.087225 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet e3783fc7fe434696898ba14ab77a9058 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.087757 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5bbaa170e90d41659516afd31247ef2d" } modify_external_catalogs: true
I20240507 08:12:13.087931 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5bbaa170e90d41659516afd31247ef2d" } modify_external_catalogs: true
I20240507 08:12:13.088821 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_75b3c3cdeef54143a244f17d717d4e17" } modify_external_catalogs: true
I20240507 08:12:13.088974 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_75b3c3cdeef54143a244f17d717d4e17" } modify_external_catalogs: true
I20240507 08:12:13.089442 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 (DEFAULT_TABLE table=default.Table_6da541994d9c4b42b19de81f931ae930 [id=2c732a9c1372437caa4db6dc50707970]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:13.089769 7290 log.cc:924] Time spent T 9211028e9eeb452d8ab5903710349ca6 P dc665657d5b34fd7aec884e0f2668e11: Append to log took a long time: real 0.164s user 0.000s sys 0.000s
I20240507 08:12:13.089788 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ec36ee6c38ab4cd29001e2a9488665c8" } new_table_name: "default.Table_12cb67bdc28944999a686f3b942d1601" modify_external_catalogs: true
I20240507 08:12:13.041452 7079 raft_consensus.cc:2258] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.100939 5656 leader_election.cc:302] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.102207 5841 raft_consensus.cc:2229] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.105190 5656 leader_election.cc:302] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.105649 6784 raft_consensus.cc:2781] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.105782 6784 raft_consensus.cc:686] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.106052 6784 consensus_queue.cc:237] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:13.118005 5843 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.531s user 0.001s sys 0.016s
I20240507 08:12:13.121376 5840 tablet_bootstrap.cc:654] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.126914 6675 raft_consensus.cc:2781] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.127172 6675 raft_consensus.cc:482] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.127259 6675 raft_consensus.cc:3037] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.130393 6675 raft_consensus.cc:504] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.131968 6675 leader_election.cc:288] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:13.143328 5378 consensus_peers.cc:485] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.143503 7297 raft_consensus.cc:482] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.143911 7296 raft_consensus.cc:482] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:13.146260 5843 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.583s user 0.001s sys 0.017s
I20240507 08:12:13.112475 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.114140 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.120180 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.125046 5839 tablet_bootstrap.cc:492] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.150853 5839 ts_tablet_manager.cc:1367] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.238s user 0.008s sys 0.032s
I20240507 08:12:13.100576 7079 ts_tablet_manager.cc:1875] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.151516 5839 raft_consensus.cc:348] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.151693 5839 raft_consensus.cc:374] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.151741 5839 raft_consensus.cc:729] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:13.151719 5377 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.008s sys 0.013s
I20240507 08:12:13.151917 5839 consensus_queue.cc:260] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.125895 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.152257 5427 tablet_service.cc:1467] Processing CreateTablet for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b (DEFAULT_TABLE table=default.Table_1633148e0c444d8eaedb7e613aa6ed3b [id=18a188c9ece44072a92126f52dd855a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.131294 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:13.153723 5654 leader_election.cc:341] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.157621 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9bae1b289994a2c8ea89be89ead6a7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.154050 5839 ts_tablet_manager.cc:1398] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240507 08:12:13.160574 5378 leader_election.cc:341] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.161352 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b0b9f85cd78748d3afce83b64b3f823b (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:12:13.162147 5408 tablet_service.cc:1467] Processing CreateTablet for tablet f748727d3a88442d90cf81bae48acd36 (DEFAULT_TABLE table=default.Table_9e717d2407ba4a54af3626573fb38c0b [id=489e7810cca5442485edf95b885b84cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.162724 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 6e2759ebfa0840f8b21d749354768c8d (DEFAULT_TABLE table=default.Table_fb6e602d297c493fa3c7969a29ea399b [id=5101edaa20304b709afaa20b4d04fc84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.160595 7297 raft_consensus.cc:504] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.163709 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.164947 7288 raft_consensus.cc:482] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.165153 7288 raft_consensus.cc:504] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:13.164958 5378 leader_election.cc:341] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.170431 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 219bad2b972c45a8930d6f47ca361858. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.171186 5377 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.552s user 0.008s sys 0.013s
I20240507 08:12:13.171764 5719 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.172477 5714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.172736 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.172760 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.173036 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.173301 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.173519 5717 raft_consensus.cc:2379] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate dc665657d5b34fd7aec884e0f2668e11 in current term 1: Already voted for candidate 1cbb887175f345ada2433adaec906fd4 in this term.
I20240507 08:12:13.176596 5839 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:13.178444 5839 tablet_bootstrap.cc:654] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.179242 5446 raft_consensus.cc:2454] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.179669 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 9b11c0fe13d34629b0491a2627f83048 (DEFAULT_TABLE table=default.Table_16de5f84b4f3402ebb29a677269ab0a0 [id=2101b18a04f74d6e8da8c7473b710775]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.181011 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c89dfe31af12466bac028b49d182b934" } modify_external_catalogs: true
I20240507 08:12:13.181190 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c89dfe31af12466bac028b49d182b934" } modify_external_catalogs: true
W20240507 08:12:13.181984 5378 leader_election.cc:341] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.183588 7288 leader_election.cc:288] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.183741 5412 tablet_service.cc:1467] Processing CreateTablet for tablet d8787ec1aec54ccaacf9096905742756 (DEFAULT_TABLE table=default.Table_c75f62ac2a774ef79d3238ef5ff7d317 [id=9abb4ed23bd04ce8a0a3b4cb758c510d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.184374 7297 leader_election.cc:288] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.184770 5843 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:13.185292 6675 raft_consensus.cc:482] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.185434 6675 raft_consensus.cc:504] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.186028 6675 leader_election.cc:288] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.186528 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:13.187076 5654 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.188935 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51fe932fe185466bbd602aea7ef70250. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.176533 7296 raft_consensus.cc:504] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.189162 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb2cb7339e6f443eb22a458c6ee9504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.190136 7294 raft_consensus.cc:482] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.191277 6880 raft_consensus.cc:482] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.191931 6880 raft_consensus.cc:504] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.192530 7294 raft_consensus.cc:504] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:13.193094 5377 leader_election.cc:341] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.193238 5377 leader_election.cc:302] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.193526 7297 raft_consensus.cc:2726] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.194494 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d23b6a487a6a4a20876771868a0d439a" } new_table_name: "default.Table_9196f7dce1f74149b05083887b83f5b0" modify_external_catalogs: true
I20240507 08:12:13.196652 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ba2e8d1e01047fd9cc1c192e9628750 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.196875 6513 ts_tablet_manager.cc:1888] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.197034 6513 log.cc:1196] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:12:13.202888 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 1578a63b4e3e4743b1057fa69653cbef (DEFAULT_TABLE table=default.Table_d3257e1057da421d901ffdb965228250 [id=531bedba63c14afb9a6d0dec9e2049a4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:13.203469 5377 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.203804 7288 consensus_queue.cc:1035] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:13.204689 5377 leader_election.cc:341] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.206383 7300 raft_consensus.cc:482] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.206522 7300 raft_consensus.cc:504] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.207185 7300 leader_election.cc:288] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.207403 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c92727908174c4d9f6aa659569027ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.207815 5377 leader_election.cc:302] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:13.208179 5517 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:12:13.208626 7288 raft_consensus.cc:2726] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:13.209434 5377 leader_election.cc:341] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.210014 5377 leader_election.cc:341] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.210412 5377 leader_election.cc:302] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.211344 7288 raft_consensus.cc:2726] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.212893 6880 leader_election.cc:288] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.213060 7294 leader_election.cc:288] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.213457 5843 ts_tablet_manager.cc:1367] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.752s user 0.005s sys 0.018s
I20240507 08:12:13.214434 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 0929c348711341d9b3529f22ff2b3df9 (DEFAULT_TABLE table=default.Table_63a2b5e8edd14d6aa283025fbf8df787 [id=a374bc2801e64de784cc8485357a8bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.214810 7296 leader_election.cc:288] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:13.215147 5378 leader_election.cc:341] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.215531 6513 ts_tablet_manager.cc:1909] T 7ba2e8d1e01047fd9cc1c192e9628750 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:13.216003 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578a63b4e3e4743b1057fa69653cbef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.216265 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de26190bc44916809856d84ad68345" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.216324 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.216424 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.216624 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.217201 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:13.217620 5654 consensus_peers.cc:485] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:13.217825 5378 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.217959 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc798eb4e02c456880ef3014ebbb7e24. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.219907 5378 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.220126 5378 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.220693 5378 leader_election.cc:341] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.220803 5378 leader_election.cc:302] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.221068 7294 raft_consensus.cc:2726] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.221817 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e2759ebfa0840f8b21d749354768c8d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.222236 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 096624b740fb4ac88a701a630c97b3eb consensus state has no leader
I20240507 08:12:13.225809 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f748727d3a88442d90cf81bae48acd36. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.229547 5840 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:13.229751 5840 ts_tablet_manager.cc:1367] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.184s user 0.003s sys 0.002s
I20240507 08:12:13.229801 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8787ec1aec54ccaacf9096905742756. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.230247 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 74c9c649ec3a4a3db414b78ef9792423. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.234671 5840 raft_consensus.cc:348] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.234879 5840 raft_consensus.cc:374] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.234944 5840 raft_consensus.cc:729] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.235302 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 957) took 3080 ms. Trace:
I20240507 08:12:13.235415 6513 rpcz_store.cc:276] 0507 08:12:10.155177 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.159094 (+ 3917us) service_pool.cc:225] Handling call
0507 08:12:13.078325 (+2919231us) ts_tablet_manager.cc:1120] Deleting tablet 7ba2e8d1e01047fd9cc1c192e9628750
0507 08:12:13.078337 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.140216 (+ 61879us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.196694 (+ 56478us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.235285 (+ 38591us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2898100}
I20240507 08:12:13.236011 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 7ba2e8d1e01047fd9cc1c192e9628750 (table default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]) successfully deleted
W20240507 08:12:13.119565 5437 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.000s sys 0.021s
I20240507 08:12:13.236219 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16b922361cc4472890fd7edbe813b667"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:13.236450 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 955) took 3081 ms. Trace:
I20240507 08:12:13.236516 6513 rpcz_store.cc:276] 0507 08:12:10.154949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.160964 (+ 6015us) service_pool.cc:225] Handling call
0507 08:12:13.236354 (+3075390us) ts_tablet_manager.cc:1120] Deleting tablet 33a425a90f0449bcaaacb4c3f653d26c
0507 08:12:13.236365 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.236434 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3055962}
W20240507 08:12:13.236791 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b922361cc4472890fd7edbe813b667 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:13.236903 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 33a425a90f0449bcaaacb4c3f653d26c because the tablet was not found. No further retry: Not found: Tablet not found: 33a425a90f0449bcaaacb4c3f653d26c
I20240507 08:12:13.119926 5841 raft_consensus.cc:2258] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.237278 6513 tablet_replica.cc:330] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:13.237531 6513 raft_consensus.cc:2229] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:13.119778 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.126704 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43f3de489904f2ba2a637d408619bfc" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.238051 6513 raft_consensus.cc:2258] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.126931 5418 tablet_service.cc:1467] Processing CreateTablet for tablet a769e920141f4ba288ebf85e3a18e791 (DEFAULT_TABLE table=default.Table_faf3a930a35b4a80aa6be69297d5ed7b [id=ff820582a283406db1942344c0ae6ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.238593 6513 ts_tablet_manager.cc:1875] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.241343 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.242874 5444 raft_consensus.cc:2454] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:13.243072 5378 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.243873 5839 tablet_bootstrap.cc:492] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.244014 5839 ts_tablet_manager.cc:1367] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.067s user 0.007s sys 0.000s
I20240507 08:12:13.244768 5839 raft_consensus.cc:348] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.244959 5839 raft_consensus.cc:374] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.245014 5839 raft_consensus.cc:729] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.245275 5839 consensus_queue.cc:260] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.246165 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0929c348711341d9b3529f22ff2b3df9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.247231 5446 raft_consensus.cc:2454] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.247287 5841 ts_tablet_manager.cc:1875] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.247437 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a769e920141f4ba288ebf85e3a18e791. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.247563 5378 leader_election.cc:302] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.252267 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c0de414a14264debbc99648f2cdabb8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.252825 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0de414a14264debbc99648f2cdabb8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.258535 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b11c0fe13d34629b0491a2627f83048. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.258587 7294 raft_consensus.cc:2726] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.268235 5839 ts_tablet_manager.cc:1398] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20240507 08:12:13.235499 5840 consensus_queue.cc:260] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.270645 5840 ts_tablet_manager.cc:1398] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20240507 08:12:13.270931 5840 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:13.273029 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7c5e5ca2e6b474f9df9ed2c59997c6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.274698 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 34adee00b529424b9f274c61ce73b0bc from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.281666 5839 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:13.284484 7101 raft_consensus.cc:482] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.284624 7101 raft_consensus.cc:504] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.285235 7101 leader_election.cc:288] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.286216 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:13.286346 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f9b60e7fd01c4c96911fff6bf62ade96 consensus state has no leader
I20240507 08:12:13.286440 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33b62a463d8f416f8407dd866c5c993b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.286695 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_42cf3e4b98c64c0db11123bb16206ebf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.287029 5654 leader_election.cc:341] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.287194 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42cf3e4b98c64c0db11123bb16206ebf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.279071 5843 raft_consensus.cc:348] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.290496 7100 raft_consensus.cc:482] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.290629 7100 raft_consensus.cc:504] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.291193 7100 leader_election.cc:288] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.291967 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:13.292650 5654 leader_election.cc:341] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.294965 5437 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.648s user 0.000s sys 0.021s
I20240507 08:12:13.298974 5839 tablet_bootstrap.cc:654] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.299875 5840 tablet_bootstrap.cc:654] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.301959 7079 ts_tablet_manager.cc:1888] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:13.302062 7079 log.cc:1196] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/f294baf04bf74dfe8500cb8c1452d8b3
I20240507 08:12:13.302358 7079 ts_tablet_manager.cc:1909] T f294baf04bf74dfe8500cb8c1452d8b3 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:13.310971 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet c1932ffe7d37455c8aa6bb83ffd11498 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.311517 5261 master_service.cc:946] ReplaceTablet: received request to replace tablet 497ab12edc724eef8d6e7b631662a4ab from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.315037 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.315353 5446 raft_consensus.cc:2454] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.315546 5439 raft_consensus.cc:3037] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.319979 7079 tablet_replica.cc:330] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:13.320053 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet f294baf04bf74dfe8500cb8c1452d8b3 (table default.Table_6d8668c1f9b94e15b7b7603a49f11a7d [id=6f1cedff6040494bb0a9bfe571648812]) successfully deleted
I20240507 08:12:13.320181 7079 raft_consensus.cc:2229] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.320356 7079 raft_consensus.cc:2258] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.320927 7079 ts_tablet_manager.cc:1875] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.322733 5439 raft_consensus.cc:2454] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.323992 5377 leader_election.cc:302] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.324100 7318 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:13.324913 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.325781 7294 raft_consensus.cc:2726] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:13.325734 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 113429us
I20240507 08:12:13.333135 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 34adee00b529424b9f274c61ce73b0bc
I20240507 08:12:13.333551 5839 tablet_bootstrap.cc:492] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.333720 5839 ts_tablet_manager.cc:1367] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.001s
I20240507 08:12:13.334038 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 34adee00b529424b9f274c61ce73b0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 4a9b0cde718548baad3ecca46e8000c9) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.335031 7322 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:13.335803 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 34adee00b529424b9f274c61ce73b0bc deleted and replaced by tablet 4a9b0cde718548baad3ecca46e8000c9
I20240507 08:12:13.336107 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c3cca313bfc24efcacf2454a66c6e114" } new_table_name: "default.Table_2b1ee92ce45f456c8b46defa7e3aa20d" modify_external_catalogs: true
I20240507 08:12:13.334482 5839 raft_consensus.cc:348] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.337335 5839 raft_consensus.cc:374] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.337401 5839 raft_consensus.cc:729] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.337646 5839 consensus_queue.cc:260] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.339624 7319 tablet.cc:1722] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:13.339824 5843 raft_consensus.cc:374] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.341724 5839 ts_tablet_manager.cc:1398] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:13.343020 5839 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:13.353302 7321 tablet.cc:1722] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:13.326817 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de26190bc44916809856d84ad68345" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.353708 5437 raft_consensus.cc:2454] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.327011 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.327219 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.355594 5843 raft_consensus.cc:729] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.355953 5843 consensus_queue.cc:260] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.356334 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069530908792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:12:13.356580 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:13.330351 6513 ts_tablet_manager.cc:1888] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.362241 6513 log.cc:1196] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/a3e58ba8c26d4f17af24eb379aca1f60
I20240507 08:12:13.334568 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 34adee00b529424b9f274c61ce73b0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 4a9b0cde718548baad3ecca46e8000c9) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.331846 5840 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:13.362632 5840 ts_tablet_manager.cc:1367] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.001s
I20240507 08:12:13.363299 5840 raft_consensus.cc:348] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.363456 5840 raft_consensus.cc:374] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.363495 5840 raft_consensus.cc:729] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.363929 5840 consensus_queue.cc:260] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.365757 5840 ts_tablet_manager.cc:1398] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:12:13.366070 5840 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:13.367518 5840 tablet_bootstrap.cc:654] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.368763 6513 ts_tablet_manager.cc:1909] T a3e58ba8c26d4f17af24eb379aca1f60 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:13.369100 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 953) took 3214 ms. Trace:
I20240507 08:12:13.369215 6513 rpcz_store.cc:276] 0507 08:12:10.154704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.161831 (+ 7127us) service_pool.cc:225] Handling call
0507 08:12:13.237231 (+3075400us) ts_tablet_manager.cc:1120] Deleting tablet a3e58ba8c26d4f17af24eb379aca1f60
0507 08:12:13.237240 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.309699 (+ 72459us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.330249 (+ 20550us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.369083 (+ 38834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3056375}
I20240507 08:12:13.369827 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet a3e58ba8c26d4f17af24eb379aca1f60 (table default.Table_d149e0b2a5ab44a2ba2f81157a507967 [id=271f1b1c669344c2ba8faf186378eac1]) successfully deleted
I20240507 08:12:13.374022 5843 ts_tablet_manager.cc:1398] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.160s user 0.001s sys 0.001s
W20240507 08:12:13.377274 5377 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.377447 5377 leader_election.cc:302] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:13.378626 5377 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.378772 5377 leader_election.cc:302] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:13.379132 5377 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.379263 5377 leader_election.cc:302] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.379484 7297 raft_consensus.cc:2726] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.379597 7297 raft_consensus.cc:2726] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.380364 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5b0f0f6f249a4ceb83421dc56c43921e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.380914 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b0f0f6f249a4ceb83421dc56c43921e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.382390 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.382969 5841 ts_tablet_manager.cc:1888] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.383196 5841 log.cc:1196] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c43f3de489904f2ba2a637d408619bfc
I20240507 08:12:13.383518 5841 ts_tablet_manager.cc:1909] T c43f3de489904f2ba2a637d408619bfc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:13.384363 5843 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:13.386538 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9501f5041dab4ca0afa406884f6c4aa1 consensus state has no leader
I20240507 08:12:13.386871 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_10a5e035c88f4438b00fb11dd74e9e4b" } new_table_name: "default.Table_90f76bdecdba45d7bb70a943d57426bd" modify_external_catalogs: true
I20240507 08:12:13.388078 6513 tablet_replica.cc:330] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:13.388325 6513 raft_consensus.cc:2229] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:13.388672 6513 raft_consensus.cc:2258] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.389263 6513 ts_tablet_manager.cc:1875] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:13.390288 5520 leader_election.cc:341] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 44ec0526e1914221bc22b3e75ea01e0d
I20240507 08:12:13.390408 5520 leader_election.cc:302] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:13.390811 5520 leader_election.cc:341] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:12:13.390895 5520 leader_election.cc:302] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.391188 7300 raft_consensus.cc:2726] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.391317 7300 raft_consensus.cc:2726] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:13.391535 5656 consensus_peers.cc:485] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44ec0526e1914221bc22b3e75ea01e0d. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.391850 5656 leader_election.cc:302] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.392165 5656 leader_election.cc:302] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.392632 7100 consensus_queue.cc:1035] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:13.392735 6784 raft_consensus.cc:2781] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.392791 6784 raft_consensus.cc:482] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.392763 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 34adee00b529424b9f274c61ce73b0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 4a9b0cde718548baad3ecca46e8000c9) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.392846 6784 raft_consensus.cc:3037] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.393003 7100 raft_consensus.cc:2781] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.393069 7100 raft_consensus.cc:482] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.393131 7100 raft_consensus.cc:3037] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.395272 6784 raft_consensus.cc:504] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.395593 7100 raft_consensus.cc:504] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.395905 6784 leader_election.cc:288] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:13.396025 7100 leader_election.cc:288] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.396608 6880 raft_consensus.cc:2726] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.396684 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.397022 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:13.397528 5654 leader_election.cc:341] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:13.397749 5654 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.408102 5843 tablet_bootstrap.cc:654] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:13.418874 5656 consensus_peers.cc:485] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.419165 5656 leader_election.cc:302] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.419474 7100 raft_consensus.cc:2781] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.419548 7100 raft_consensus.cc:482] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.419613 7100 raft_consensus.cc:3037] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.422252 7100 raft_consensus.cc:504] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.384171 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 873) took 5977 ms. Trace:
I20240507 08:12:13.423046 5841 rpcz_store.cc:276] 0507 08:12:07.406357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.407641 (+ 1284us) service_pool.cc:225] Handling call
0507 08:12:12.893321 (+5485680us) ts_tablet_manager.cc:1120] Deleting tablet c43f3de489904f2ba2a637d408619bfc
0507 08:12:12.893367 (+ 46us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.257384 (+364017us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.346229 (+ 88845us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.384153 (+ 37924us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164296,"tablet-delete.queue_time_us":5485429}
I20240507 08:12:13.423555 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c43f3de489904f2ba2a637d408619bfc (table default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]) successfully deleted
I20240507 08:12:13.423606 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.423628 5841 tablet_replica.cc:330] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:13.423861 5841 raft_consensus.cc:2229] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:13.424041 5654 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.424223 7100 leader_election.cc:288] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:13.424000 5841 raft_consensus.cc:2258] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.429433 5841 ts_tablet_manager.cc:1875] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.432493 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.434928 5447 raft_consensus.cc:2454] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.437228 5839 tablet_bootstrap.cc:654] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.441325 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:13.448869 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5bbebe0430784c14affcd57084473b5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.449440 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bbebe0430784c14affcd57084473b5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.450276 5843 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:13.451066 5843 ts_tablet_manager.cc:1367] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.002s
I20240507 08:12:13.451771 5843 raft_consensus.cc:348] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.452181 5843 raft_consensus.cc:374] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.452309 5843 raft_consensus.cc:729] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.452574 5843 consensus_queue.cc:260] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.451954 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 4d588afa11864c11b32837c2fee3e1e3 (DEFAULT_TABLE table=default.Table_fb97573b1b0e4931be83172a373e0592 [id=f02a66fd4602448096db31a632c1ed6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.453140 5843 ts_tablet_manager.cc:1398] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:13.453289 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 4d588afa11864c11b32837c2fee3e1e3 (DEFAULT_TABLE table=default.Table_fb97573b1b0e4931be83172a373e0592 [id=f02a66fd4602448096db31a632c1ed6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.453387 5843 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:13.454669 5843 tablet_bootstrap.cc:654] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.455155 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2754fc46c36346f9828198e3bb6ed3a7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.455330 5443 raft_consensus.cc:3037] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.455461 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a56cb45381604ac9a69e6f3edaa4eefc" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.455551 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet fa932c6ad48f4ac183aabba8699cb148 (table default.Table_12cb67bdc28944999a686f3b942d1601 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677]). Expected version 5 got 4
I20240507 08:12:13.455698 5447 raft_consensus.cc:3037] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.455909 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
I20240507 08:12:13.456076 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 3 got 0
I20240507 08:12:13.456281 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:13.456440 5257 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_a63ec02af6ab4e85830582d44267f8c1 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 1 got 0
I20240507 08:12:13.457856 5443 raft_consensus.cc:2454] T 2754fc46c36346f9828198e3bb6ed3a7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.458142 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d588afa11864c11b32837c2fee3e1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.458616 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbe72f7b400456cb091375050f16e2d (DEFAULT_TABLE table=default.Table_bba5a7409437414987e560b41b2c8875 [id=426825c7761445169b9a5611dacf0103]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.459028 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbe72f7b400456cb091375050f16e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.459661 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d588afa11864c11b32837c2fee3e1e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.461117 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5471c74c33cd42e8aaf47f71f61adff7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.461272 5437 raft_consensus.cc:3037] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.463706 5437 raft_consensus.cc:2454] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.465440 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e23516b85a2c422d8f09edaa41f0d52a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.466106 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e23516b85a2c422d8f09edaa41f0d52a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.467144 5656 leader_election.cc:302] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.468592 6675 raft_consensus.cc:2781] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.468726 6675 raft_consensus.cc:482] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.468844 6675 raft_consensus.cc:3037] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.470685 5447 raft_consensus.cc:2454] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.471362 6675 raft_consensus.cc:504] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.472009 6675 leader_election.cc:288] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.472173 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 4d588afa11864c11b32837c2fee3e1e3 (DEFAULT_TABLE table=default.Table_fb97573b1b0e4931be83172a373e0592 [id=f02a66fd4602448096db31a632c1ed6f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:13.472306 5520 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 81490.1us
I20240507 08:12:13.472545 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d588afa11864c11b32837c2fee3e1e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.472582 5520 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:13.473007 5520 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.473259 7300 raft_consensus.cc:482] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.473356 7300 raft_consensus.cc:504] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.473881 7300 leader_election.cc:288] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.473989 7300 raft_consensus.cc:2726] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.474539 5520 leader_election.cc:302] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.474793 7300 raft_consensus.cc:2726] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:13.475131 5520 leader_election.cc:341] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:12:13.475265 5520 leader_election.cc:302] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.475593 7300 raft_consensus.cc:2726] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.477982 5840 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:13.478164 5840 ts_tablet_manager.cc:1367] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.003s
I20240507 08:12:13.478876 5840 raft_consensus.cc:348] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.479110 5840 raft_consensus.cc:374] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.479208 5840 raft_consensus.cc:729] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.479445 5840 consensus_queue.cc:260] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.479925 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.480016 5840 ts_tablet_manager.cc:1398] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:13.480255 5840 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:13.480500 5654 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.481161 5656 leader_election.cc:302] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.481596 5656 leader_election.cc:302] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.482075 7100 raft_consensus.cc:2781] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.482232 7100 raft_consensus.cc:686] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.482506 7100 consensus_queue.cc:237] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.483495 6784 raft_consensus.cc:2781] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.483574 6784 raft_consensus.cc:482] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.483530 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 90d19086b15c4c028e058d94cb4afc9d (DEFAULT_TABLE table=default.Table_7079f1efc8a149a9b81f0b7667492b6b [id=16602a9366a74e1085c6798d2010defb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.483644 6784 raft_consensus.cc:3037] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.483876 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d19086b15c4c028e058d94cb4afc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.478328 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.483168 5425 tablet_service.cc:1467] Processing CreateTablet for tablet d718aa8fe99041688964d1a05cef49a0 (DEFAULT_TABLE table=default.Table_e9828e7d20ab4c959f6963420d4fa85e [id=989e47a7ed07439c9fe6aa2cb0be744b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.486081 6784 raft_consensus.cc:504] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.486384 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718aa8fe99041688964d1a05cef49a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.486562 6784 leader_election.cc:288] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.490902 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbe72f7b400456cb091375050f16e2d (DEFAULT_TABLE table=default.Table_bba5a7409437414987e560b41b2c8875 [id=426825c7761445169b9a5611dacf0103]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.491117 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.491243 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbe72f7b400456cb091375050f16e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.492275 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 6d368875a22f4883ae9fd0e19f1ae997 (DEFAULT_TABLE table=default.Table_ad88688ccc4f46edaafaf3b8c6e3f8ec [id=d1f18f0538c74b109c44d279e7282121]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.492595 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d368875a22f4883ae9fd0e19f1ae997. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.492722 5654 leader_election.cc:341] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.492859 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 90d19086b15c4c028e058d94cb4afc9d (DEFAULT_TABLE table=default.Table_7079f1efc8a149a9b81f0b7667492b6b [id=16602a9366a74e1085c6798d2010defb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.493157 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d19086b15c4c028e058d94cb4afc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.493849 5698 tablet_service.cc:1467] Processing CreateTablet for tablet d718aa8fe99041688964d1a05cef49a0 (DEFAULT_TABLE table=default.Table_e9828e7d20ab4c959f6963420d4fa85e [id=989e47a7ed07439c9fe6aa2cb0be744b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.494217 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718aa8fe99041688964d1a05cef49a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.495510 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.497866 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 90d19086b15c4c028e058d94cb4afc9d (DEFAULT_TABLE table=default.Table_7079f1efc8a149a9b81f0b7667492b6b [id=16602a9366a74e1085c6798d2010defb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.498163 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d19086b15c4c028e058d94cb4afc9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.501336 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.485416 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 2dbe72f7b400456cb091375050f16e2d (DEFAULT_TABLE table=default.Table_bba5a7409437414987e560b41b2c8875 [id=426825c7761445169b9a5611dacf0103]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.501691 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2dbe72f7b400456cb091375050f16e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.501802 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 6d368875a22f4883ae9fd0e19f1ae997 (DEFAULT_TABLE table=default.Table_ad88688ccc4f46edaafaf3b8c6e3f8ec [id=d1f18f0538c74b109c44d279e7282121]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.502103 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d368875a22f4883ae9fd0e19f1ae997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.503177 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d718aa8fe99041688964d1a05cef49a0 (DEFAULT_TABLE table=default.Table_e9828e7d20ab4c959f6963420d4fa85e [id=989e47a7ed07439c9fe6aa2cb0be744b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.503474 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d718aa8fe99041688964d1a05cef49a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.510892 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 6d368875a22f4883ae9fd0e19f1ae997 (DEFAULT_TABLE table=default.Table_ad88688ccc4f46edaafaf3b8c6e3f8ec [id=d1f18f0538c74b109c44d279e7282121]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.511478 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d368875a22f4883ae9fd0e19f1ae997. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.513769 5413 tablet_service.cc:1467] Processing CreateTablet for tablet dcd7ec455826497893af43763c7e7acf (DEFAULT_TABLE table=default.Table_adbdf44307174a5b8a7a916c5fc6c5bb [id=1496a6e54248484bb617fc171fa24b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.514124 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd7ec455826497893af43763c7e7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.514122 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 56771302b1654642b9eaa21e459065ee (DEFAULT_TABLE table=default.Table_27ec04cf1e6643b496d0d1a979e3f4f9 [id=dad9bacec2b049a89352570a38ff5a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.514468 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56771302b1654642b9eaa21e459065ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.514928 5693 tablet_service.cc:1467] Processing CreateTablet for tablet dcd7ec455826497893af43763c7e7acf (DEFAULT_TABLE table=default.Table_adbdf44307174a5b8a7a916c5fc6c5bb [id=1496a6e54248484bb617fc171fa24b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.515293 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd7ec455826497893af43763c7e7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.518435 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 56771302b1654642b9eaa21e459065ee (DEFAULT_TABLE table=default.Table_27ec04cf1e6643b496d0d1a979e3f4f9 [id=dad9bacec2b049a89352570a38ff5a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.519039 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 1da3535b598e4609b4a1d4a3c1e3da95 (DEFAULT_TABLE table=default.Table_d0aa0582569b459bbed495999da97539 [id=63fb9d4844294452a39bafd3ad0f5aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.519363 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da3535b598e4609b4a1d4a3c1e3da95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.519572 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56771302b1654642b9eaa21e459065ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.521508 5548 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39e5e0eef462da07b8e7a0eb508b9 (DEFAULT_TABLE table=default.Table_200c77e7e235497c952d6a523e6a30e0 [id=25afcdb941a0471bba6a60cb37fda8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.521914 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39e5e0eef462da07b8e7a0eb508b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.522572 5681 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39e5e0eef462da07b8e7a0eb508b9 (DEFAULT_TABLE table=default.Table_200c77e7e235497c952d6a523e6a30e0 [id=25afcdb941a0471bba6a60cb37fda8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.522886 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39e5e0eef462da07b8e7a0eb508b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.526113 5563 tablet_service.cc:1467] Processing CreateTablet for tablet dcd7ec455826497893af43763c7e7acf (DEFAULT_TABLE table=default.Table_adbdf44307174a5b8a7a916c5fc6c5bb [id=1496a6e54248484bb617fc171fa24b4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.526486 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd7ec455826497893af43763c7e7acf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.531842 5546 tablet_service.cc:1467] Processing CreateTablet for tablet c6d972e15fa840668e74ac1416c2e2d9 (DEFAULT_TABLE table=default.Table_8e2edbe2bb52425bbde255742f34f784 [id=f9f89e9e94e44e8a89d268f490b9b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.532337 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 434db44a626e426aaa2d4d055899a56b (DEFAULT_TABLE table=default.Table_2468219608d6479fba6a3ef7216bd1a0 [id=30d5a8d8645b407aae6d294efdcae390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.532784 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434db44a626e426aaa2d4d055899a56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.532868 5564 tablet_service.cc:1467] Processing CreateTablet for tablet dcff9560b48849c5b9524d009b178177 (DEFAULT_TABLE table=default.Table_99f001a5e47f44928eb0d9d255bd7423 [id=0350ecc71b514feaadac7bac86133702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.533387 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcff9560b48849c5b9524d009b178177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.533603 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d972e15fa840668e74ac1416c2e2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.535655 6513 ts_tablet_manager.cc:1888] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.539503 5423 tablet_service.cc:1467] Processing CreateTablet for tablet c6d972e15fa840668e74ac1416c2e2d9 (DEFAULT_TABLE table=default.Table_8e2edbe2bb52425bbde255742f34f784 [id=f9f89e9e94e44e8a89d268f490b9b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.539834 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d972e15fa840668e74ac1416c2e2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.540205 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 434db44a626e426aaa2d4d055899a56b (DEFAULT_TABLE table=default.Table_2468219608d6479fba6a3ef7216bd1a0 [id=30d5a8d8645b407aae6d294efdcae390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.540553 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434db44a626e426aaa2d4d055899a56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.536170 6784 consensus_queue.cc:1035] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:13.536438 5656 leader_election.cc:302] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.542737 6784 raft_consensus.cc:2781] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.542837 6784 raft_consensus.cc:482] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.542943 6784 raft_consensus.cc:3037] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.545379 6784 raft_consensus.cc:504] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.546164 6784 leader_election.cc:288] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.546466 6513 log.cc:1196] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/d6f8f4205f1f48abaf4fcd7113ad44a8
I20240507 08:12:13.546926 6513 ts_tablet_manager.cc:1909] T d6f8f4205f1f48abaf4fcd7113ad44a8 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:13.542574 5656 consensus_peers.cc:485] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.548139 5656 leader_election.cc:302] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.549065 7100 raft_consensus.cc:2781] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.549244 7100 raft_consensus.cc:686] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.549340 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.549608 7100 consensus_queue.cc:237] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.550205 5691 tablet_service.cc:1467] Processing CreateTablet for tablet c6d972e15fa840668e74ac1416c2e2d9 (DEFAULT_TABLE table=default.Table_8e2edbe2bb52425bbde255742f34f784 [id=f9f89e9e94e44e8a89d268f490b9b34c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.550420 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 434db44a626e426aaa2d4d055899a56b (DEFAULT_TABLE table=default.Table_2468219608d6479fba6a3ef7216bd1a0 [id=30d5a8d8645b407aae6d294efdcae390]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.518666 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 56771302b1654642b9eaa21e459065ee (DEFAULT_TABLE table=default.Table_27ec04cf1e6643b496d0d1a979e3f4f9 [id=dad9bacec2b049a89352570a38ff5a0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.551204 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56771302b1654642b9eaa21e459065ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.551785 6675 raft_consensus.cc:482] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.551977 6675 raft_consensus.cc:504] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.552626 6675 leader_election.cc:288] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.553314 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 1da3535b598e4609b4a1d4a3c1e3da95 (DEFAULT_TABLE table=default.Table_d0aa0582569b459bbed495999da97539 [id=63fb9d4844294452a39bafd3ad0f5aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.553822 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 1da3535b598e4609b4a1d4a3c1e3da95 (DEFAULT_TABLE table=default.Table_d0aa0582569b459bbed495999da97539 [id=63fb9d4844294452a39bafd3ad0f5aba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.554244 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da3535b598e4609b4a1d4a3c1e3da95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.555884 5840 tablet_bootstrap.cc:654] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.557225 5656 leader_election.cc:302] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.557538 7101 raft_consensus.cc:2781] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.557700 7101 raft_consensus.cc:686] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:13.557809 5654 leader_election.cc:341] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.557930 7101 consensus_queue.cc:237] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.558374 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:13.558880 5654 leader_election.cc:341] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.561339 7101 consensus_queue.cc:1035] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:13.562489 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 902b50c7dc084276a1968f08e7355679 consensus state has no leader
I20240507 08:12:13.567729 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
W20240507 08:12:13.568943 5378 consensus_peers.cc:485] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba7c1aaf55d74c979ec1fc0a57f15f44. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:13.570663 5520 leader_election.cc:341] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:12:13.571691 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6d972e15fa840668e74ac1416c2e2d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.574326 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 434db44a626e426aaa2d4d055899a56b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.576099 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1e1cfba5e8234af8823ff12bdefb9723"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.576627 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e1cfba5e8234af8823ff12bdefb9723 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.579404 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1da3535b598e4609b4a1d4a3c1e3da95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.582104 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 972) took 3211 ms. Trace:
I20240507 08:12:13.582192 6513 rpcz_store.cc:276] 0507 08:12:10.370872 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.371561 (+ 689us) service_pool.cc:225] Handling call
0507 08:12:13.388025 (+3016464us) ts_tablet_manager.cc:1120] Deleting tablet d6f8f4205f1f48abaf4fcd7113ad44a8
0507 08:12:13.388036 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.431396 (+ 43360us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.535569 (+104173us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.582068 (+ 46499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3016223}
I20240507 08:12:13.582353 7294 raft_consensus.cc:482] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.582504 6513 tablet_replica.cc:330] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:13.582476 7294 raft_consensus.cc:504] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.582677 6513 raft_consensus.cc:2229] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:13.582859 6513 pending_rounds.cc:62] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:13.582890 6513 pending_rounds.cc:68] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924805926490112 op_type: NO_OP noop_request { }
I20240507 08:12:13.582976 6513 raft_consensus.cc:2866] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:13.583041 6513 raft_consensus.cc:2258] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.583036 7294 leader_election.cc:288] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.584605 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.584676 5656 leader_election.cc:302] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:13.585194 5378 leader_election.cc:341] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.585414 7333 raft_consensus.cc:2781] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.585552 7333 raft_consensus.cc:686] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.585836 7333 consensus_queue.cc:237] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.586081 7318 tablet.cc:1722] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:13.589773 5409 tablet_service.cc:1467] Processing CreateTablet for tablet dcff9560b48849c5b9524d009b178177 (DEFAULT_TABLE table=default.Table_99f001a5e47f44928eb0d9d255bd7423 [id=0350ecc71b514feaadac7bac86133702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.590154 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcff9560b48849c5b9524d009b178177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.591400 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 3f30af566be44760828e1672d83fc903 (DEFAULT_TABLE table=default.Table_45f5c047857b4e9fbcdc45084bdf09a4 [id=67d81a3853c1447296fb8ffe9fbe7c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.591728 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f30af566be44760828e1672d83fc903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.594015 5839 tablet_bootstrap.cc:492] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.594208 5839 ts_tablet_manager.cc:1367] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.251s user 0.006s sys 0.003s
I20240507 08:12:13.594888 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.595614 5691 tablet_service.cc:1467] Processing CreateTablet for tablet dcff9560b48849c5b9524d009b178177 (DEFAULT_TABLE table=default.Table_99f001a5e47f44928eb0d9d255bd7423 [id=0350ecc71b514feaadac7bac86133702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.595986 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcff9560b48849c5b9524d009b178177. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.596007 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (table default.Table_87bb488658cd4819a51ff1ff0993ceab [id=799497aabd9b4f17bc42b655e947b3ed]) successfully deleted
W20240507 08:12:13.596334 5377 leader_election.cc:341] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.596503 5377 leader_election.cc:302] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.596905 7294 raft_consensus.cc:2726] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.596892 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 3f30af566be44760828e1672d83fc903 (DEFAULT_TABLE table=default.Table_45f5c047857b4e9fbcdc45084bdf09a4 [id=67d81a3853c1447296fb8ffe9fbe7c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.597208 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f30af566be44760828e1672d83fc903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.597381 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 08e0045140d14a4e8810cc428ab1d2f0 (DEFAULT_TABLE table=default.Table_c1bf63edee4348b1aad5f47c7fe5b726 [id=5993e26cf53f41d0b1cadf9f3945185f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.597765 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e0045140d14a4e8810cc428ab1d2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.597980 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 08e0045140d14a4e8810cc428ab1d2f0 (DEFAULT_TABLE table=default.Table_c1bf63edee4348b1aad5f47c7fe5b726 [id=5993e26cf53f41d0b1cadf9f3945185f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.598251 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 497ab12edc724eef8d6e7b631662a4ab
I20240507 08:12:13.598286 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e0045140d14a4e8810cc428ab1d2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.598723 5261 catalog_manager.cc:6554] ReplaceTablet: tablet 497ab12edc724eef8d6e7b631662a4ab deleted and replaced by tablet 2b79d55827864491a6eddac99e383ba8
I20240507 08:12:13.601347 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bbe582048e334931902cc697661690ca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:13.603098 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 497ab12edc724eef8d6e7b631662a4ab with delete_type TABLET_DATA_DELETED (replaced by tablet 2b79d55827864491a6eddac99e383ba8) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.603375 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35e34b60b78b402faf51e003c97db787" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.603484 6513 ts_tablet_manager.cc:1875] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.603821 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 497ab12edc724eef8d6e7b631662a4ab with delete_type TABLET_DATA_DELETED (replaced by tablet 2b79d55827864491a6eddac99e383ba8) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.603905 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 497ab12edc724eef8d6e7b631662a4ab with delete_type TABLET_DATA_DELETED (replaced by tablet 2b79d55827864491a6eddac99e383ba8) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.605120 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.605388 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 52e9c1158e27465280e1043d91917e3e
I20240507 08:12:13.606163 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ee23488ce5a940559de6005aa58093f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.606619 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee23488ce5a940559de6005aa58093f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.607184 5839 raft_consensus.cc:348] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.607390 5839 raft_consensus.cc:374] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.607442 5839 raft_consensus.cc:729] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.607642 5839 consensus_queue.cc:260] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.608441 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9c1158e27465280e1043d91917e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.608981 5423 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9c1158e27465280e1043d91917e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.610507 5843 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:13.610623 5843 ts_tablet_manager.cc:1367] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.157s user 0.009s sys 0.009s
I20240507 08:12:13.610579 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.610749 5439 raft_consensus.cc:3037] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.611320 5843 raft_consensus.cc:348] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.611578 5843 raft_consensus.cc:374] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.611677 5843 raft_consensus.cc:729] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:13.611745 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bbe582048e334931902cc697661690ca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.611937 5843 consensus_queue.cc:260] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.612466 5843 ts_tablet_manager.cc:1398] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:13.612751 5843 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:13.613366 5439 raft_consensus.cc:2454] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.613888 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd4379b6fded4a52843f55f27ae67121" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.614136 5843 tablet_bootstrap.cc:654] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.614114 5447 raft_consensus.cc:3037] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.618589 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 52e9c1158e27465280e1043d91917e3e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.623174 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 74c9c649ec3a4a3db414b78ef9792423
I20240507 08:12:13.623667 5839 ts_tablet_manager.cc:1398] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.029s user 0.000s sys 0.001s
I20240507 08:12:13.623716 5263 master_service.cc:946] ReplaceTablet: received request to replace tablet 4c62499ccee34036b23e4fe0804fd2ae from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.625104 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fec60780cac3409fb0c1d6d61f8a8e25" } new_table_name: "default.Table_45da47d5031f45938e714a8403e5539c" modify_external_catalogs: true
I20240507 08:12:13.626505 5517 leader_election.cc:302] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.627605 5447 raft_consensus.cc:2454] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.627712 5839 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:13.627838 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.629129 7329 raft_consensus.cc:2726] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.629436 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.629962 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.630429 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 08e0045140d14a4e8810cc428ab1d2f0 (DEFAULT_TABLE table=default.Table_c1bf63edee4348b1aad5f47c7fe5b726 [id=5993e26cf53f41d0b1cadf9f3945185f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.630743 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.630833 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08e0045140d14a4e8810cc428ab1d2f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.630895 5446 raft_consensus.cc:3037] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.633203 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:13.633273 5446 raft_consensus.cc:2454] T ba28c36f91e141f4a1372e7544b44281 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.636346 7345 tablet.cc:1722] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:13.637061 5546 tablet_service.cc:1467] Processing CreateTablet for tablet a40a4e36cc6145cf8311e06720cbd4c2 (DEFAULT_TABLE table=default.Table_8cf67af06cf44e458ec7c70c6a18ffd0 [id=d234dc0c202349ba9fa49f860a603456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.637435 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a4e36cc6145cf8311e06720cbd4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.637763 5839 tablet_bootstrap.cc:654] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.638222 5683 tablet_service.cc:1467] Processing CreateTablet for tablet a40a4e36cc6145cf8311e06720cbd4c2 (DEFAULT_TABLE table=default.Table_8cf67af06cf44e458ec7c70c6a18ffd0 [id=d234dc0c202349ba9fa49f860a603456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.638222 5549 tablet_service.cc:1467] Processing CreateTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.638289 5690 tablet_service.cc:1467] Processing CreateTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.638528 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a4e36cc6145cf8311e06720cbd4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.638574 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68badb3b5644ed8b007b2b04bc4f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.638782 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68badb3b5644ed8b007b2b04bc4f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.637451 5413 tablet_service.cc:1467] Processing CreateTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 (DEFAULT_TABLE table=default.Table_2e53530be7694370bbab56e5fdc054c0 [id=2b1def56287847cc974afa8fb2daf2e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.642323 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 3f30af566be44760828e1672d83fc903 (DEFAULT_TABLE table=default.Table_45f5c047857b4e9fbcdc45084bdf09a4 [id=67d81a3853c1447296fb8ffe9fbe7c2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.642637 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f30af566be44760828e1672d83fc903. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.646505 7100 raft_consensus.cc:482] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.646709 7100 raft_consensus.cc:504] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.647497 7100 leader_election.cc:288] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.642014 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68badb3b5644ed8b007b2b04bc4f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.650751 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9cf92632da4643cfab913cd925fccef9
I20240507 08:12:13.653030 5839 tablet_bootstrap.cc:492] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.654027 5839 ts_tablet_manager.cc:1367] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.003s
I20240507 08:12:13.654628 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf92632da4643cfab913cd925fccef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.654760 5422 tablet_service.cc:1467] Processing CreateTablet for tablet f0f39e5e0eef462da07b8e7a0eb508b9 (DEFAULT_TABLE table=default.Table_200c77e7e235497c952d6a523e6a30e0 [id=25afcdb941a0471bba6a60cb37fda8ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.648661 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 74c9c649ec3a4a3db414b78ef9792423 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:12 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.655606 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eecaf7ddbbe044fe86665d7b0fe5cfa2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:13.655817 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:13.656109 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eecaf7ddbbe044fe86665d7b0fe5cfa2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.658764 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf92632da4643cfab913cd925fccef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:13.659629 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 8d1a4e05273b4f8c83dc8c7825a39310 (table default.Table_6a769d7d87b84f1da5328510e326530f [id=819ac00611cb44e2bb3cf13a221e522b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8d1a4e05273b4f8c83dc8c7825a39310 consensus state has no leader
I20240507 08:12:13.660374 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.660418 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 9cf92632da4643cfab913cd925fccef9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.661708 5839 raft_consensus.cc:348] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.663581 5839 raft_consensus.cc:374] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.664211 5839 raft_consensus.cc:729] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:13.664701 5654 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.666206 5839 consensus_queue.cc:260] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.670462 5410 tablet_service.cc:1467] Processing CreateTablet for tablet d8e03f8655714d309d5c8ae49c4b314b (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.671906 5546 tablet_service.cc:1467] Processing CreateTablet for tablet d8e03f8655714d309d5c8ae49c4b314b (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.673089 5554 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed28b1c6c74077b3ba267d8fd69b2d (DEFAULT_TABLE table=default.Table_4b785d2845eb44679d48de42c13e1bdb [id=81473f93f049450197a9d338686c519d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.673946 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e03f8655714d309d5c8ae49c4b314b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.675072 5417 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed28b1c6c74077b3ba267d8fd69b2d (DEFAULT_TABLE table=default.Table_4b785d2845eb44679d48de42c13e1bdb [id=81473f93f049450197a9d338686c519d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.675969 5689 tablet_service.cc:1467] Processing CreateTablet for tablet d8e03f8655714d309d5c8ae49c4b314b (DEFAULT_TABLE table=default.Table_910aab0a8d6a4249b5e8a0ae88916c9f [id=fe0ce55b371d4214a8c9bf3679bbc14e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.676371 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e03f8655714d309d5c8ae49c4b314b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.677786 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed28b1c6c74077b3ba267d8fd69b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.677820 5685 tablet_service.cc:1467] Processing CreateTablet for tablet e9ed28b1c6c74077b3ba267d8fd69b2d (DEFAULT_TABLE table=default.Table_4b785d2845eb44679d48de42c13e1bdb [id=81473f93f049450197a9d338686c519d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.678151 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed28b1c6c74077b3ba267d8fd69b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.680974 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c8aaabc3e26140fda6f86ce0e518b8f5
I20240507 08:12:13.681627 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dd9261ef80574699827814799f609a4d" } new_table_name: "default.Table_6ef7238dc88a40b1ad4bc8a78079ec26" modify_external_catalogs: true
I20240507 08:12:13.683562 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.684192 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0f39e5e0eef462da07b8e7a0eb508b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.685355 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8e03f8655714d309d5c8ae49c4b314b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.686316 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9ed28b1c6c74077b3ba267d8fd69b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.689882 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:13.690073 5444 raft_consensus.cc:2454] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.690676 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.691265 5420 tablet_service.cc:1467] Processing CreateTablet for tablet a40a4e36cc6145cf8311e06720cbd4c2 (DEFAULT_TABLE table=default.Table_8cf67af06cf44e458ec7c70c6a18ffd0 [id=d234dc0c202349ba9fa49f860a603456]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.691682 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a40a4e36cc6145cf8311e06720cbd4c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.695724 5585 raft_consensus.cc:3037] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.696290 5841 ts_tablet_manager.cc:1888] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.696381 5841 log.cc:1196] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/35e34b60b78b402faf51e003c97db787
I20240507 08:12:13.696731 5841 ts_tablet_manager.cc:1909] T 35e34b60b78b402faf51e003c97db787 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:13.697021 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 872) took 6290 ms. Trace:
I20240507 08:12:13.697086 5841 rpcz_store.cc:276] 0507 08:12:07.406250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.409630 (+ 3380us) service_pool.cc:225] Handling call
0507 08:12:13.423572 (+6013942us) ts_tablet_manager.cc:1120] Deleting tablet 35e34b60b78b402faf51e003c97db787
0507 08:12:13.423582 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.482133 (+ 58551us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.696215 (+214082us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.697004 (+ 789us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6013787}
I20240507 08:12:13.697443 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 35e34b60b78b402faf51e003c97db787 (table default.Table_7007a63c36ea47c78769b5bcafec4cf9 [id=71fd821e13734824a24f76fa2c220f1a]) successfully deleted
I20240507 08:12:13.697742 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 875) took 6249 ms. Trace:
I20240507 08:12:13.697861 5841 rpcz_store.cc:276] 0507 08:12:07.448169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.452568 (+ 4399us) service_pool.cc:225] Handling call
0507 08:12:13.697520 (+6244952us) ts_tablet_manager.cc:1120] Deleting tablet dfd9624f4337437bb11b247048e28c08
0507 08:12:13.697547 (+ 27us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.697704 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6244739}
W20240507 08:12:13.698354 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet dfd9624f4337437bb11b247048e28c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfd9624f4337437bb11b247048e28c08 already in progress: creating tablet
I20240507 08:12:13.698493 5841 tablet_replica.cc:330] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:13.706449 7349 tablet.cc:1722] T 2d45d93ed0d34806b9d9451c101f10af P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:13.719244 5656 leader_election.cc:302] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.719805 6675 raft_consensus.cc:2781] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.719915 6675 raft_consensus.cc:686] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.720141 6675 consensus_queue.cc:237] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.723553 5843 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:13.723520 5656 leader_election.cc:302] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.723760 5843 ts_tablet_manager.cc:1367] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.005s
I20240507 08:12:13.723878 5656 leader_election.cc:302] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.724105 5546 tablet_service.cc:1467] Processing CreateTablet for tablet c3556038f04447cd9a04a9133999b125 (DEFAULT_TABLE table=default.Table_3dbb85e2d6814888a9680e5bd557124d [id=b211bff880ce41268fd767a3d5ae7c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.724463 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3556038f04447cd9a04a9133999b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.725855 7101 raft_consensus.cc:2781] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.725992 7101 raft_consensus.cc:686] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.726197 7101 consensus_queue.cc:237] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.726800 5564 tablet_service.cc:1467] Processing CreateTablet for tablet ececf80512074304ab59db6cbb83f00b (DEFAULT_TABLE table=default.Table_6bd4fbf9fea6459893b5a20cf44ff554 [id=5105359ce2df4d3eabe84b0e5c1f5743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.727100 6784 raft_consensus.cc:2781] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.727147 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ececf80512074304ab59db6cbb83f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.727201 6784 raft_consensus.cc:686] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.727375 6784 consensus_queue.cc:237] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.729321 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 95b72048b1b04004901a25a4d00e2b2d (DEFAULT_TABLE table=default.Table_7d1b03d5425d44c695c58cb9cfa6f68c [id=061a81e9378846fe9c12b3c5240a2d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.729691 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b72048b1b04004901a25a4d00e2b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.729630 7330 consensus_queue.cc:1035] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:13.730192 5656 leader_election.cc:341] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 79524df2415146ab926425c42068626c
I20240507 08:12:13.730736 5656 leader_election.cc:302] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.731395 7330 raft_consensus.cc:2726] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.735460 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 95b72048b1b04004901a25a4d00e2b2d (DEFAULT_TABLE table=default.Table_7d1b03d5425d44c695c58cb9cfa6f68c [id=061a81e9378846fe9c12b3c5240a2d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.735610 5843 raft_consensus.cc:348] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.735832 5843 raft_consensus.cc:374] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.735935 5843 raft_consensus.cc:729] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.736024 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b72048b1b04004901a25a4d00e2b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.736184 5843 consensus_queue.cc:260] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:13.736807 5843 ts_tablet_manager.cc:1398] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:13.736888 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d07221497ce7468fb86c257492cef397
I20240507 08:12:13.737820 5843 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:13.739293 5843 tablet_bootstrap.cc:654] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.733289 5427 tablet_service.cc:1467] Processing CreateTablet for tablet c3556038f04447cd9a04a9133999b125 (DEFAULT_TABLE table=default.Table_3dbb85e2d6814888a9680e5bd557124d [id=b211bff880ce41268fd767a3d5ae7c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.733688 7345 tablet.cc:1722] T fa932c6ad48f4ac183aabba8699cb148 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:13.732981 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet c8aaabc3e26140fda6f86ce0e518b8f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.740988 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3556038f04447cd9a04a9133999b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.741657 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_756b5368f96f42b98537f8bca05456ec" } new_table_name: "default.Table_c07eca3b27ff427aa91ecefdb785f27b" modify_external_catalogs: true
I20240507 08:12:13.743235 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 95b72048b1b04004901a25a4d00e2b2d (DEFAULT_TABLE table=default.Table_7d1b03d5425d44c695c58cb9cfa6f68c [id=061a81e9378846fe9c12b3c5240a2d7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.743933 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b72048b1b04004901a25a4d00e2b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.744555 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet d07221497ce7468fb86c257492cef397 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.737428 5841 raft_consensus.cc:2229] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.744904 5841 raft_consensus.cc:2258] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.745152 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet d07221497ce7468fb86c257492cef397 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.745607 5841 ts_tablet_manager.cc:1875] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.748327 5689 tablet_service.cc:1467] Processing CreateTablet for tablet c3556038f04447cd9a04a9133999b125 (DEFAULT_TABLE table=default.Table_3dbb85e2d6814888a9680e5bd557124d [id=b211bff880ce41268fd767a3d5ae7c16]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.748742 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3556038f04447cd9a04a9133999b125. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.750238 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet d07221497ce7468fb86c257492cef397 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:13.751067 5654 consensus_peers.cc:485] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.759244 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_12cb67bdc28944999a686f3b942d1601 [id=e72c5d8cfeb14c3b9ae7d8c6e1499677] alter complete (version 5)
I20240507 08:12:13.759799 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f46b82cdb45a45d9886098ab730da4df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:13.760257 5840 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
W20240507 08:12:13.760306 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f46b82cdb45a45d9886098ab730da4df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.760377 5840 ts_tablet_manager.cc:1367] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.280s user 0.002s sys 0.001s
I20240507 08:12:13.761106 5840 raft_consensus.cc:348] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.761281 5840 raft_consensus.cc:374] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.761338 5840 raft_consensus.cc:729] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.761504 5840 consensus_queue.cc:260] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.762138 5840 ts_tablet_manager.cc:1398] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:13.762428 5840 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:13.766613 5840 tablet_bootstrap.cc:654] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.770256 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 97f9487af6a84e2c83a76bfcdcdd4c47 (DEFAULT_TABLE table=default.Table_d7b00c2cd3f84b2ba1f5c18b18d93bb2 [id=dc8a0ed8c9484f0aa8e45ea5e917b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.770658 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f9487af6a84e2c83a76bfcdcdd4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.772163 6513 ts_tablet_manager.cc:1888] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.772408 6513 log.cc:1196] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:13.772855 6513 ts_tablet_manager.cc:1909] T aa094773f3bf418492bcd5b5021fd429 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:13.773216 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 971) took 3402 ms. Trace:
I20240507 08:12:13.773861 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d270dcfe84128a31fe48d5f2be090 (DEFAULT_TABLE table=default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.774221 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d270dcfe84128a31fe48d5f2be090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.774325 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 97f9487af6a84e2c83a76bfcdcdd4c47 (DEFAULT_TABLE table=default.Table_d7b00c2cd3f84b2ba1f5c18b18d93bb2 [id=dc8a0ed8c9484f0aa8e45ea5e917b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.774762 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f9487af6a84e2c83a76bfcdcdd4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.781549 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 3524e293f4bb4741b4efb5bffe054767 (DEFAULT_TABLE table=default.Table_8f309bfc29fa40b39a9b836aff0d30ea [id=08d3f31917654fcb86f671a409bb450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.784224 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3524e293f4bb4741b4efb5bffe054767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.785524 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d270dcfe84128a31fe48d5f2be090 (DEFAULT_TABLE table=default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.785879 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d270dcfe84128a31fe48d5f2be090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.786837 5258 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:13.789892 5413 tablet_service.cc:1467] Processing CreateTablet for tablet ececf80512074304ab59db6cbb83f00b (DEFAULT_TABLE table=default.Table_6bd4fbf9fea6459893b5a20cf44ff554 [id=5105359ce2df4d3eabe84b0e5c1f5743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.790494 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ececf80512074304ab59db6cbb83f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.792035 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_29d81546e07145b6aad46145e0f37793"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.792704 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29d81546e07145b6aad46145e0f37793 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.796773 5585 raft_consensus.cc:3037] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.802320 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 3524e293f4bb4741b4efb5bffe054767 (DEFAULT_TABLE table=default.Table_8f309bfc29fa40b39a9b836aff0d30ea [id=08d3f31917654fcb86f671a409bb450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.803820 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3524e293f4bb4741b4efb5bffe054767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.804065 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d8af63f7647643fd9c9eced3f9c15be0 (DEFAULT_TABLE table=default.Table_6e0d718bf25e458d8090be71389a3c89 [id=171015a52d8a4267858f73bcba4d47be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.806494 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af63f7647643fd9c9eced3f9c15be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.778441 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 3524e293f4bb4741b4efb5bffe054767 (DEFAULT_TABLE table=default.Table_8f309bfc29fa40b39a9b836aff0d30ea [id=08d3f31917654fcb86f671a409bb450d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.807543 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3524e293f4bb4741b4efb5bffe054767. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.779042 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 97f9487af6a84e2c83a76bfcdcdd4c47 (DEFAULT_TABLE table=default.Table_d7b00c2cd3f84b2ba1f5c18b18d93bb2 [id=dc8a0ed8c9484f0aa8e45ea5e917b033]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.813450 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97f9487af6a84e2c83a76bfcdcdd4c47. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.813303 5422 tablet_service.cc:1467] Processing CreateTablet for tablet d8af63f7647643fd9c9eced3f9c15be0 (DEFAULT_TABLE table=default.Table_6e0d718bf25e458d8090be71389a3c89 [id=171015a52d8a4267858f73bcba4d47be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.779454 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 8f9d270dcfe84128a31fe48d5f2be090 (DEFAULT_TABLE table=default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.817974 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f9d270dcfe84128a31fe48d5f2be090. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.817025 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af63f7647643fd9c9eced3f9c15be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.820590 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_40e14999737746f0984e6386706a3d9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:13.821125 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40e14999737746f0984e6386706a3d9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.779723 5681 tablet_service.cc:1467] Processing CreateTablet for tablet ececf80512074304ab59db6cbb83f00b (DEFAULT_TABLE table=default.Table_6bd4fbf9fea6459893b5a20cf44ff554 [id=5105359ce2df4d3eabe84b0e5c1f5743]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.822022 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ececf80512074304ab59db6cbb83f00b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.779990 5698 tablet_service.cc:1467] Processing CreateTablet for tablet d8af63f7647643fd9c9eced3f9c15be0 (DEFAULT_TABLE table=default.Table_6e0d718bf25e458d8090be71389a3c89 [id=171015a52d8a4267858f73bcba4d47be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.825371 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8af63f7647643fd9c9eced3f9c15be0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.836107 6513 rpcz_store.cc:276] 0507 08:12:10.370711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.373711 (+ 3000us) service_pool.cc:225] Handling call
0507 08:12:13.582463 (+3208752us) ts_tablet_manager.cc:1120] Deleting tablet aa094773f3bf418492bcd5b5021fd429
0507 08:12:13.582474 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.705506 (+123032us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.772065 (+ 66559us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.773195 (+ 1130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3208575}
I20240507 08:12:13.836206 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6fe4ab85c9476a9d9b224e8b99736d (DEFAULT_TABLE table=default.Table_5eb952438aa148978bdecbc42cb74318 [id=61c1fd5cf85c42e9ab897d8c554c4e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.836575 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6fe4ab85c9476a9d9b224e8b99736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.836695 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 973) took 3437 ms. Trace:
I20240507 08:12:13.836876 6513 rpcz_store.cc:276] 0507 08:12:10.399097 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.399160 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:13.836562 (+3437402us) ts_tablet_manager.cc:1120] Deleting tablet 9ccbe2e368d14b56859d62d1cc742f90
0507 08:12:13.836571 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.836678 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3437205}
I20240507 08:12:13.836927 5427 tablet_service.cc:1467] Processing CreateTablet for tablet c2d61f0374ef49a4a5daca2a488f5874 (DEFAULT_TABLE table=default.Table_d0672e8f460d4a779e4defb93cbbff2d [id=7f72e47c43874b048647b6cc3b49087a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.837234 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d61f0374ef49a4a5daca2a488f5874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.837293 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 983) took 3181 ms. Trace:
I20240507 08:12:13.837366 6513 rpcz_store.cc:276] 0507 08:12:10.655966 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.656019 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:13.837195 (+3181176us) ts_tablet_manager.cc:1120] Deleting tablet 81fcf82e60c94e9eaadd75bd37595261
0507 08:12:13.837203 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.837277 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180867}
I20240507 08:12:13.838405 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet aa094773f3bf418492bcd5b5021fd429 (table default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]) successfully deleted
W20240507 08:12:13.838618 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9ccbe2e368d14b56859d62d1cc742f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccbe2e368d14b56859d62d1cc742f90 already in progress: creating tablet
W20240507 08:12:13.838816 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 81fcf82e60c94e9eaadd75bd37595261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81fcf82e60c94e9eaadd75bd37595261 already in progress: creating tablet
I20240507 08:12:13.839076 5409 tablet_service.cc:1467] Processing CreateTablet for tablet e6582d8301f74443abc3b336ace9a466 (DEFAULT_TABLE table=default.Table_ebe94534f9de4a8eaf2d035236c832dc [id=dfe128a81f664241a9a7d50a8ac9f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.839195 5683 tablet_service.cc:1467] Processing CreateTablet for tablet c2d61f0374ef49a4a5daca2a488f5874 (DEFAULT_TABLE table=default.Table_d0672e8f460d4a779e4defb93cbbff2d [id=7f72e47c43874b048647b6cc3b49087a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.839354 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6582d8301f74443abc3b336ace9a466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.839390 6675 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dc665657d5b34fd7aec884e0f2668e11)
I20240507 08:12:13.839497 6675 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.840070 6675 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.840809 5691 tablet_service.cc:1467] Processing CreateTablet for tablet e219304ab92343689b0b0fb9fffccb1f (DEFAULT_TABLE table=default.Table_f1c86fd506164b62a5000e951232875e [id=aee20d6a02ae40eeb072d4bd8c832325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.841192 5416 tablet_service.cc:1467] Processing CreateTablet for tablet e219304ab92343689b0b0fb9fffccb1f (DEFAULT_TABLE table=default.Table_f1c86fd506164b62a5000e951232875e [id=aee20d6a02ae40eeb072d4bd8c832325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.841555 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e219304ab92343689b0b0fb9fffccb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.842053 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4c62499ccee34036b23e4fe0804fd2ae
I20240507 08:12:13.842559 5263 catalog_manager.cc:6554] ReplaceTablet: tablet 4c62499ccee34036b23e4fe0804fd2ae deleted and replaced by tablet 24e3ae77a7704423946b9074c19cb30d
I20240507 08:12:13.842809 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 24e3ae77a7704423946b9074c19cb30d) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:13.842897 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b59f41c55cab461abd20c0fc9ae984bd" } new_table_name: "default.Table_ea12b1a814f546a19315986dde83e3db" modify_external_catalogs: true
I20240507 08:12:13.843822 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8159bc51194247b29d795f76d944f971 (DEFAULT_TABLE table=default.Table_3a4888c1005a4a4783fa36c86a79c2f3 [id=50bb5fbf978848ee996a5d5c8b433448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.844372 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 24e3ae77a7704423946b9074c19cb30d) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:13.844691 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159bc51194247b29d795f76d944f971. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.853080 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 84263.2us
I20240507 08:12:13.853710 5656 leader_election.cc:302] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.854038 7100 raft_consensus.cc:2781] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:13.854143 7100 raft_consensus.cc:482] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:13.854250 7100 raft_consensus.cc:3037] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.854287 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.854542 5583 raft_consensus.cc:2454] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:13.855211 5563 tablet_service.cc:1467] Processing CreateTablet for tablet e6582d8301f74443abc3b336ace9a466 (DEFAULT_TABLE table=default.Table_ebe94534f9de4a8eaf2d035236c832dc [id=dfe128a81f664241a9a7d50a8ac9f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.856637 7100 raft_consensus.cc:504] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.857597 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c62499ccee34036b23e4fe0804fd2ae with delete_type TABLET_DATA_DELETED (replaced by tablet 24e3ae77a7704423946b9074c19cb30d) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.858001 5554 tablet_service.cc:1467] Processing CreateTablet for tablet c2d61f0374ef49a4a5daca2a488f5874 (DEFAULT_TABLE table=default.Table_d0672e8f460d4a779e4defb93cbbff2d [id=7f72e47c43874b048647b6cc3b49087a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:13.858096 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a96a6023c78f4d1196b3d2fcf2f22329 consensus state has no leader
I20240507 08:12:13.858121 7100 leader_election.cc:288] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.858289 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6582d8301f74443abc3b336ace9a466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.858491 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 8159bc51194247b29d795f76d944f971 (DEFAULT_TABLE table=default.Table_3a4888c1005a4a4783fa36c86a79c2f3 [id=50bb5fbf978848ee996a5d5c8b433448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.858522 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2cd819bfa63841308bf0554cd623e25a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:13.858803 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159bc51194247b29d795f76d944f971. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.859015 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd819bfa63841308bf0554cd623e25a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:13.859393 5654 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:12:13.860108 7100 raft_consensus.cc:2781] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240507 08:12:13.860231 7100 raft_consensus.cc:482] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dc665657d5b34fd7aec884e0f2668e11)
I20240507 08:12:13.860361 7100 raft_consensus.cc:3037] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Advancing to term 2
I20240507 08:12:13.861042 5564 tablet_service.cc:1467] Processing CreateTablet for tablet e219304ab92343689b0b0fb9fffccb1f (DEFAULT_TABLE table=default.Table_f1c86fd506164b62a5000e951232875e [id=aee20d6a02ae40eeb072d4bd8c832325]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.861349 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e219304ab92343689b0b0fb9fffccb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.862283 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.862859 7100 raft_consensus.cc:504] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.863518 7100 leader_election.cc:288] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.865142 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d61f0374ef49a4a5daca2a488f5874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.865325 7294 raft_consensus.cc:482] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.865422 5840 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:13.865525 5840 ts_tablet_manager.cc:1367] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.002s
I20240507 08:12:13.865446 7294 raft_consensus.cc:504] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.866047 7294 leader_election.cc:288] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:13.866245 5840 raft_consensus.cc:348] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.866488 5840 raft_consensus.cc:374] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.866583 5840 raft_consensus.cc:729] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.866852 5840 consensus_queue.cc:260] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.867367 5840 ts_tablet_manager.cc:1398] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240507 08:12:13.868162 5654 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.868623 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1037c3c0af5e464bb23a311e552fcf21" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:13.868783 5584 raft_consensus.cc:3037] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 2
I20240507 08:12:13.868953 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.869030 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 984) took 3208 ms. Trace:
I20240507 08:12:13.869046 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:13.869112 6513 rpcz_store.cc:276] 0507 08:12:10.660974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.661024 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:13.837561 (+3176537us) ts_tablet_manager.cc:1120] Deleting tablet 39e5e67df91f4029bf55e5a82dc31f82
0507 08:12:13.868835 (+ 31274us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.869013 (+ 178us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3175682}
W20240507 08:12:13.869484 5378 leader_election.cc:341] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.870559 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6fe4ab85c9476a9d9b224e8b99736d (DEFAULT_TABLE table=default.Table_5eb952438aa148978bdecbc42cb74318 [id=61c1fd5cf85c42e9ab897d8c554c4e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.870918 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6fe4ab85c9476a9d9b224e8b99736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.871563 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e219304ab92343689b0b0fb9fffccb1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.874440 5584 raft_consensus.cc:2454] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 2.
I20240507 08:12:13.877290 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d61f0374ef49a4a5daca2a488f5874. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.886408 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "831153d2b89e43f8b08b086f7a8f3a54" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:13.886595 5444 raft_consensus.cc:3037] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.889218 5444 raft_consensus.cc:2454] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:13.897964 5839 ts_tablet_manager.cc:1398] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.243s user 0.002s sys 0.000s
I20240507 08:12:13.898377 5839 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:13.899938 5839 tablet_bootstrap.cc:654] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.907160 5841 ts_tablet_manager.cc:1888] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.907456 5693 tablet_service.cc:1467] Processing CreateTablet for tablet e6582d8301f74443abc3b336ace9a466 (DEFAULT_TABLE table=default.Table_ebe94534f9de4a8eaf2d035236c832dc [id=dfe128a81f664241a9a7d50a8ac9f13e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.907933 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6582d8301f74443abc3b336ace9a466. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.916486 5654 leader_election.cc:302] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:12:13.917066 7358 raft_consensus.cc:2781] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 2 FOLLOWER]: Leader election won for term 2
I20240507 08:12:13.917292 7358 raft_consensus.cc:686] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 2 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.919356 5264 catalog_manager.cc:5484] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:13.920663 5264 catalog_manager.cc:5484] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:13.921538 5264 catalog_manager.cc:5484] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
W20240507 08:12:13.924798 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 39e5e67df91f4029bf55e5a82dc31f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39e5e67df91f4029bf55e5a82dc31f82 already in progress: creating tablet
I20240507 08:12:13.924703 7358 consensus_queue.cc:237] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.926060 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 985) took 3264 ms. Trace:
I20240507 08:12:13.926173 6513 rpcz_store.cc:276] 0507 08:12:10.661512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.661562 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:13.869341 (+3207779us) ts_tablet_manager.cc:1120] Deleting tablet 924d8d2fe2f148cba13f107e4c70fc68
0507 08:12:13.925959 (+ 56618us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.926047 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207444}
W20240507 08:12:13.927009 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fafc295e1cc04cf6a64e594e9bad69b2 consensus state has no leader
I20240507 08:12:13.927294 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 7e26c7ef6f2845cabe09d09045d4feda from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.927449 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 986) took 3265 ms. Trace:
I20240507 08:12:13.928359 6513 rpcz_store.cc:276] 0507 08:12:10.662358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.662401 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:13.926995 (+3264594us) ts_tablet_manager.cc:1120] Deleting tablet 128c2034d917473eb5c496319fdc9625
0507 08:12:13.927005 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.927435 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3264453}
I20240507 08:12:13.929019 6513 tablet_replica.cc:330] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:13.929775 6513 raft_consensus.cc:2229] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:13.926613 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6fe4ab85c9476a9d9b224e8b99736d (DEFAULT_TABLE table=default.Table_5eb952438aa148978bdecbc42cb74318 [id=61c1fd5cf85c42e9ab897d8c554c4e39]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.931355 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6fe4ab85c9476a9d9b224e8b99736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.925559 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 8159bc51194247b29d795f76d944f971 (DEFAULT_TABLE table=default.Table_3a4888c1005a4a4783fa36c86a79c2f3 [id=50bb5fbf978848ee996a5d5c8b433448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.934731 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8159bc51194247b29d795f76d944f971. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.915014 5377 leader_election.cc:341] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:13.937727 5377 leader_election.cc:302] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.938195 7294 raft_consensus.cc:2726] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.938544 5841 log.cc:1196] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/0d617213b959452ab6519809e56db128
W20240507 08:12:13.938917 5656 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:13.938984 5841 ts_tablet_manager.cc:1909] T 0d617213b959452ab6519809e56db128 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:13.939843 5656 leader_election.cc:302] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:13.940687 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 874) took 6528 ms. Trace:
I20240507 08:12:13.940834 5841 rpcz_store.cc:276] 0507 08:12:07.412692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.412756 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:13.698425 (+6285669us) ts_tablet_manager.cc:1120] Deleting tablet 0d617213b959452ab6519809e56db128
0507 08:12:13.698434 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.800935 (+102501us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.907039 (+106104us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.940668 (+ 33629us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":678,"tablet-delete.queue_time_us":6243853}
I20240507 08:12:13.941303 5843 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:13.941319 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 0d617213b959452ab6519809e56db128 (table default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6 [id=bce0bd66e7b74db7934937410c81b20b]) successfully deleted
I20240507 08:12:13.941478 5843 ts_tablet_manager.cc:1367] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.204s user 0.022s sys 0.043s
I20240507 08:12:13.941813 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 877) took 6415 ms. Trace:
I20240507 08:12:13.941941 5841 rpcz_store.cc:276] 0507 08:12:07.526210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.529348 (+ 3138us) service_pool.cc:225] Handling call
0507 08:12:13.941694 (+6412346us) ts_tablet_manager.cc:1120] Deleting tablet 5aa521dd21634cd0af25652931069c41
0507 08:12:13.941702 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.941798 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6412106}
I20240507 08:12:13.942183 5843 raft_consensus.cc:348] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.942349 5843 raft_consensus.cc:374] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:13.942368 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5aa521dd21634cd0af25652931069c41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5aa521dd21634cd0af25652931069c41 already in progress: creating tablet
I20240507 08:12:13.942394 5843 raft_consensus.cc:729] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.942593 5843 consensus_queue.cc:260] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.943018 5843 ts_tablet_manager.cc:1398] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240507 08:12:13.943233 5843 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:13.943253 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 888) took 6288 ms. Trace:
I20240507 08:12:13.943409 5841 rpcz_store.cc:276] 0507 08:12:07.654708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.654770 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:13.942878 (+6288108us) ts_tablet_manager.cc:1120] Deleting tablet fccc127d380d4035905c1b152ab1017b
0507 08:12:13.942885 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.942953 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6287778}
W20240507 08:12:13.943830 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet fccc127d380d4035905c1b152ab1017b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fccc127d380d4035905c1b152ab1017b already in progress: creating tablet
I20240507 08:12:13.944242 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 889) took 6280 ms. Trace:
I20240507 08:12:13.944350 5841 rpcz_store.cc:276] 0507 08:12:07.663955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.664010 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:13.944097 (+6280087us) ts_tablet_manager.cc:1120] Deleting tablet 32ed504c4cf443e48ab285466b9e9b5d
0507 08:12:13.944106 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.944227 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6279922}
I20240507 08:12:13.944659 5843 tablet_bootstrap.cc:654] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:13.944882 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 32ed504c4cf443e48ab285466b9e9b5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32ed504c4cf443e48ab285466b9e9b5d already in progress: creating tablet
I20240507 08:12:13.945552 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 890) took 6270 ms. Trace:
I20240507 08:12:13.945712 5841 rpcz_store.cc:276] 0507 08:12:07.675231 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:07.675309 (+ 78us) service_pool.cc:225] Handling call
0507 08:12:13.945426 (+6270117us) ts_tablet_manager.cc:1120] Deleting tablet bc029bc110b847f79d876ec3ab568832
0507 08:12:13.945434 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.945535 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6269949}
W20240507 08:12:13.946112 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bc029bc110b847f79d876ec3ab568832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc029bc110b847f79d876ec3ab568832 already in progress: creating tablet
I20240507 08:12:13.946472 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 908) took 5694 ms. Trace:
I20240507 08:12:13.946941 5841 rpcz_store.cc:276] 0507 08:12:08.252324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.259819 (+ 7495us) service_pool.cc:225] Handling call
0507 08:12:13.946371 (+5686552us) ts_tablet_manager.cc:1120] Deleting tablet 8005100f64fd49a3ac29d34aa728fdfb
0507 08:12:13.946379 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.946456 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5686272}
I20240507 08:12:13.947314 5584 raft_consensus.cc:3037] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:13.948150 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 (DEFAULT_TABLE table=default.Table_e4cc0dc4d852467c9bdeaeb440199da8 [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.948345 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 (DEFAULT_TABLE table=default.Table_e4cc0dc4d852467c9bdeaeb440199da8 [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.948621 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e779c5c97c44e5bcc7c0eb0194fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.948705 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e779c5c97c44e5bcc7c0eb0194fd95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.948328 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 910) took 5669 ms. Trace:
I20240507 08:12:13.951898 5841 rpcz_store.cc:276] 0507 08:12:08.278316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.282178 (+ 3862us) service_pool.cc:225] Handling call
0507 08:12:13.947518 (+5665340us) ts_tablet_manager.cc:1120] Deleting tablet c40ed1afee294143ab85a8c03d080b74
0507 08:12:13.947526 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.947604 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5664877}
W20240507 08:12:13.952308 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c40ed1afee294143ab85a8c03d080b74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c40ed1afee294143ab85a8c03d080b74 already in progress: creating tablet
I20240507 08:12:13.952673 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 911) took 5652 ms. Trace:
I20240507 08:12:13.952777 5841 rpcz_store.cc:276] 0507 08:12:08.300426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.305130 (+ 4704us) service_pool.cc:225] Handling call
0507 08:12:13.952569 (+5647439us) ts_tablet_manager.cc:1120] Deleting tablet c7bb96bb9d8f4930b531a3dd30d733c8
0507 08:12:13.952577 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.952659 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5647148}
W20240507 08:12:13.953261 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c7bb96bb9d8f4930b531a3dd30d733c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7bb96bb9d8f4930b531a3dd30d733c8 already in progress: creating tablet
I20240507 08:12:13.953735 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 909) took 5686 ms. Trace:
I20240507 08:12:13.953843 5841 rpcz_store.cc:276] 0507 08:12:08.267324 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.307951 (+ 40627us) service_pool.cc:225] Handling call
0507 08:12:13.953621 (+5645670us) ts_tablet_manager.cc:1120] Deleting tablet a570e39a52d84d3dbcaeb9ea005d9f41
0507 08:12:13.953629 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.953721 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5645399}
W20240507 08:12:13.954295 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a570e39a52d84d3dbcaeb9ea005d9f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a570e39a52d84d3dbcaeb9ea005d9f41 already in progress: creating tablet
I20240507 08:12:13.954591 5841 tablet_replica.cc:330] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:13.954826 5841 raft_consensus.cc:2229] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.955688 5841 raft_consensus.cc:2258] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.956322 5841 ts_tablet_manager.cc:1875] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.955464 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30b345f9a4979b17af3c3d71f6f54 (DEFAULT_TABLE table=default.Table_a9c53f96ea0f4ef2b792fbf2ee71a1e0 [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.958155 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30b345f9a4979b17af3c3d71f6f54. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:13.959774 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 690b258a9b564ed1b07aef29a774b749 consensus state has no leader
I20240507 08:12:13.960180 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b17576c2e21b405380ccce99ebbd7b7e" } new_table_name: "default.Table_77f1b8f4de134bf8acda3315c2f08293" modify_external_catalogs: true
I20240507 08:12:13.961393 5256 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a] alter complete (version 2)
I20240507 08:12:13.961771 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 2d913dc032b44a4fbab59b42efcba58c from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:13.963725 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e4cc0dc4d852467c9bdeaeb440199da8" } modify_external_catalogs: true
I20240507 08:12:13.963883 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e4cc0dc4d852467c9bdeaeb440199da8" } modify_external_catalogs: true
W20240507 08:12:13.967097 5656 leader_election.cc:341] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:13.940099 6513 raft_consensus.cc:2258] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.959007 5840 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:13.968426 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:13.968837 5840 tablet_bootstrap.cc:654] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:13.970136 6784 raft_consensus.cc:2781] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:13.970244 6784 raft_consensus.cc:686] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:13.970439 6784 consensus_queue.cc:237] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:13.974045 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 924d8d2fe2f148cba13f107e4c70fc68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924d8d2fe2f148cba13f107e4c70fc68 already in progress: creating tablet
I20240507 08:12:13.974530 6513 ts_tablet_manager.cc:1875] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.976670 5841 ts_tablet_manager.cc:1888] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:13.977789 5841 log.cc:1196] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/28e5a380f2784e3a806afcbb5cd2519a
W20240507 08:12:13.978916 5654 consensus_peers.cc:485] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:13.979599 5841 ts_tablet_manager.cc:1909] T 28e5a380f2784e3a806afcbb5cd2519a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:13.979537 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30b345f9a4979b17af3c3d71f6f54 (DEFAULT_TABLE table=default.Table_a9c53f96ea0f4ef2b792fbf2ee71a1e0 [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.979904 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30b345f9a4979b17af3c3d71f6f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.980590 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 918) took 5617 ms. Trace:
I20240507 08:12:13.981009 5841 rpcz_store.cc:276] 0507 08:12:08.363510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.363908 (+ 398us) service_pool.cc:225] Handling call
0507 08:12:13.954549 (+5590641us) ts_tablet_manager.cc:1120] Deleting tablet 28e5a380f2784e3a806afcbb5cd2519a
0507 08:12:13.954557 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.967729 (+ 13172us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.976565 (+ 8836us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:13.980571 (+ 4006us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5590419}
I20240507 08:12:13.981730 5841 tablet_replica.cc:330] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:13.981973 5841 raft_consensus.cc:2229] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:13.982182 5841 raft_consensus.cc:2258] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:13.982777 5841 ts_tablet_manager.cc:1875] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:13.981498 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 28e5a380f2784e3a806afcbb5cd2519a (table default.Table_8a58f4e04c4147368d43d162c9613c9f [id=0e33fa8f8fff4e09916ca2783a4903e7]) successfully deleted
W20240507 08:12:13.985471 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 128c2034d917473eb5c496319fdc9625 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c2034d917473eb5c496319fdc9625 already in progress: creating tablet
I20240507 08:12:13.986704 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 82c30b345f9a4979b17af3c3d71f6f54 (DEFAULT_TABLE table=default.Table_a9c53f96ea0f4ef2b792fbf2ee71a1e0 [id=e797752d15e34220b083a7ca8e4a715e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:13.987128 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 82c30b345f9a4979b17af3c3d71f6f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:13.989773 7329 raft_consensus.cc:482] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.989929 7329 raft_consensus.cc:504] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.990679 7329 leader_election.cc:288] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:13.991276 7329 raft_consensus.cc:482] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:13.991389 7329 raft_consensus.cc:504] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.991914 7329 leader_election.cc:288] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:13.992226 5520 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:12:13.992394 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:13.994338 5517 leader_election.cc:341] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:12:13.994443 5517 leader_election.cc:302] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.994659 7329 raft_consensus.cc:2726] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.995461 5839 tablet_bootstrap.cc:492] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:13.995527 5840 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:13.995664 5840 ts_tablet_manager.cc:1367] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.000s
I20240507 08:12:13.995683 5839 ts_tablet_manager.cc:1367] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.097s user 0.001s sys 0.003s
I20240507 08:12:13.996327 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba277be774d32912d1174bec68b51" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:13.996448 5839 raft_consensus.cc:348] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.996600 5839 raft_consensus.cc:374] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.996701 5839 raft_consensus.cc:729] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.996815 5517 leader_election.cc:302] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:13.997021 7329 raft_consensus.cc:2726] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:13.996975 5839 consensus_queue.cc:260] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.998627 5840 raft_consensus.cc:348] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:13.998811 5840 raft_consensus.cc:374] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:13.998870 5840 raft_consensus.cc:729] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:13.999061 5840 consensus_queue.cc:260] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.000077 5843 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.000985 5843 ts_tablet_manager.cc:1367] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.000s
I20240507 08:12:14.001770 5840 ts_tablet_manager.cc:1398] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:14.000838 5841 ts_tablet_manager.cc:1888] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.005973 5841 log.cc:1196] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e298f11ff9f54e48b8de435f6849f39a
I20240507 08:12:14.006430 5841 ts_tablet_manager.cc:1909] T e298f11ff9f54e48b8de435f6849f39a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.006759 5840 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.007609 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 919) took 5644 ms. Trace:
I20240507 08:12:14.009107 5840 tablet_bootstrap.cc:654] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.009222 5841 rpcz_store.cc:276] 0507 08:12:08.363623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.364434 (+ 811us) service_pool.cc:225] Handling call
0507 08:12:13.981686 (+5617252us) ts_tablet_manager.cc:1120] Deleting tablet e298f11ff9f54e48b8de435f6849f39a
0507 08:12:13.981695 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.986638 (+ 4943us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.000751 (+ 14113us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.007587 (+ 6836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5617075}
I20240507 08:12:14.009707 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e298f11ff9f54e48b8de435f6849f39a (table default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]) successfully deleted
I20240507 08:12:14.009779 5841 tablet_replica.cc:330] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.009958 5841 raft_consensus.cc:2229] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.010116 5841 raft_consensus.cc:2258] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.010623 5841 ts_tablet_manager.cc:1875] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.021996 5841 ts_tablet_manager.cc:1888] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.023406 6513 ts_tablet_manager.cc:1888] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.023383 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70c8dabdda144b77a238abeacaa20e6e" } modify_external_catalogs: true
I20240507 08:12:14.023519 6513 log.cc:1196] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:14.023598 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_70c8dabdda144b77a238abeacaa20e6e" } modify_external_catalogs: true
I20240507 08:12:14.023896 6513 ts_tablet_manager.cc:1909] T b0b9f85cd78748d3afce83b64b3f823b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.026198 5841 log.cc:1196] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b6e68364238c4b66b8c0de0f4a2ea855
I20240507 08:12:14.027190 5841 ts_tablet_manager.cc:1909] T b6e68364238c4b66b8c0de0f4a2ea855 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.027572 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 987) took 3320 ms. Trace:
I20240507 08:12:14.027683 6513 rpcz_store.cc:276] 0507 08:12:10.706714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.706771 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:13.928973 (+3222202us) ts_tablet_manager.cc:1120] Deleting tablet b0b9f85cd78748d3afce83b64b3f823b
0507 08:12:13.928984 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.995684 (+ 66700us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.023310 (+ 27626us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.027552 (+ 4242us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14653,"tablet-delete.queue_time_us":3221999}
I20240507 08:12:14.028079 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b0b9f85cd78748d3afce83b64b3f823b (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:12:14.029512 6513 tablet_replica.cc:330] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.030112 6513 raft_consensus.cc:2229] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:14.031862 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 920) took 5668 ms. Trace:
I20240507 08:12:14.032394 6513 pending_rounds.cc:62] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:14.032464 6513 pending_rounds.cc:68] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924795867938816 op_type: NO_OP noop_request { }
I20240507 08:12:14.032582 6513 raft_consensus.cc:2866] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:14.032683 6513 raft_consensus.cc:2258] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.033061 5841 rpcz_store.cc:276] 0507 08:12:08.363747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.369697 (+ 5950us) service_pool.cc:225] Handling call
0507 08:12:14.009732 (+5640035us) ts_tablet_manager.cc:1120] Deleting tablet b6e68364238c4b66b8c0de0f4a2ea855
0507 08:12:14.009743 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.019129 (+ 9386us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.021914 (+ 2785us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.031842 (+ 9928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5639833}
I20240507 08:12:14.033207 6513 ts_tablet_manager.cc:1875] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.033972 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 934) took 5353 ms. Trace:
I20240507 08:12:14.033991 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b6e68364238c4b66b8c0de0f4a2ea855 (table default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]) successfully deleted
I20240507 08:12:14.034389 5841 rpcz_store.cc:276] 0507 08:12:08.680422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.699685 (+ 19263us) service_pool.cc:225] Handling call
0507 08:12:14.033826 (+5334141us) ts_tablet_manager.cc:1120] Deleting tablet 182124e13ca749c8a2a96d38514158b1
0507 08:12:14.033836 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.033957 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5333967}
W20240507 08:12:14.035063 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 182124e13ca749c8a2a96d38514158b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 182124e13ca749c8a2a96d38514158b1 already in progress: creating tablet
I20240507 08:12:14.035499 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 936) took 5354 ms. Trace:
I20240507 08:12:14.035635 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 (DEFAULT_TABLE table=default.Table_e4cc0dc4d852467c9bdeaeb440199da8 [id=e3fb663ef2f44f9f838494aa1561e9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.035913 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 58d9a185726144d589e20ce85c4897fa (DEFAULT_TABLE table=default.Table_4c3768696f85440290e9da3a265d45d6 [id=3020a441b8d4461f9a2961dde8fd3d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.035912 5841 rpcz_store.cc:276] 0507 08:12:08.680732 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.700181 (+ 19449us) service_pool.cc:225] Handling call
0507 08:12:14.035393 (+5335212us) ts_tablet_manager.cc:1120] Deleting tablet d2240fd14155495a8c97a0af96090faf
0507 08:12:14.035401 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.035484 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5335073}
I20240507 08:12:14.036063 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11e779c5c97c44e5bcc7c0eb0194fd95. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:14.036250 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d2240fd14155495a8c97a0af96090faf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2240fd14155495a8c97a0af96090faf already in progress: creating tablet
I20240507 08:12:14.036818 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 937) took 5356 ms. Trace:
I20240507 08:12:14.037153 5841 rpcz_store.cc:276] 0507 08:12:08.680817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.700324 (+ 19507us) service_pool.cc:225] Handling call
0507 08:12:14.036723 (+5336399us) ts_tablet_manager.cc:1120] Deleting tablet 8fc65070bbae4db19c5d14b6670d599c
0507 08:12:14.036732 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.036803 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5336294}
W20240507 08:12:14.037856 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8fc65070bbae4db19c5d14b6670d599c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fc65070bbae4db19c5d14b6670d599c already in progress: creating tablet
I20240507 08:12:14.038889 5564 tablet_service.cc:1467] Processing CreateTablet for tablet b65e251f3d5c4414afe413a54a519d85 (DEFAULT_TABLE table=default.Table_af114d78455749ce9330a056f6505e04 [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.038949 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 58d9a185726144d589e20ce85c4897fa (DEFAULT_TABLE table=default.Table_4c3768696f85440290e9da3a265d45d6 [id=3020a441b8d4461f9a2961dde8fd3d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.039404 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d9a185726144d589e20ce85c4897fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:14.041137 5695 tablet_service.cc:1467] Processing CreateTablet for tablet b65e251f3d5c4414afe413a54a519d85 (DEFAULT_TABLE table=default.Table_af114d78455749ce9330a056f6505e04 [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.041450 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65e251f3d5c4414afe413a54a519d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:14.041863 5413 tablet_service.cc:1467] Processing CreateTablet for tablet b65e251f3d5c4414afe413a54a519d85 (DEFAULT_TABLE table=default.Table_af114d78455749ce9330a056f6505e04 [id=513cac3f33c8405183c2af4f13522541]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.042176 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65e251f3d5c4414afe413a54a519d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:14.046563 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 58d9a185726144d589e20ce85c4897fa (DEFAULT_TABLE table=default.Table_4c3768696f85440290e9da3a265d45d6 [id=3020a441b8d4461f9a2961dde8fd3d7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:14.046919 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d9a185726144d589e20ce85c4897fa. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:14.050153 5656 consensus_peers.cc:485] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 28e5a380f2784e3a806afcbb5cd2519a. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.058030 5839 ts_tablet_manager.cc:1398] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.062s user 0.000s sys 0.001s
I20240507 08:12:14.059356 7079 ts_tablet_manager.cc:1888] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:14.059567 7079 log.cc:1196] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4de7053cfaea4a54a402c9d826613c05
I20240507 08:12:14.060045 7079 ts_tablet_manager.cc:1909] T 4de7053cfaea4a54a402c9d826613c05 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:14.064285 5840 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.064415 5840 ts_tablet_manager.cc:1367] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.004s
I20240507 08:12:14.065176 5840 raft_consensus.cc:348] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.065368 5840 raft_consensus.cc:374] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.065441 5840 raft_consensus.cc:729] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.065670 5840 consensus_queue.cc:260] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.066205 5840 ts_tablet_manager.cc:1398] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:14.066448 5840 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.067803 5840 tablet_bootstrap.cc:654] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.071543 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 938) took 5390 ms. Trace:
I20240507 08:12:14.071698 5841 rpcz_store.cc:276] 0507 08:12:08.680907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.700431 (+ 19524us) service_pool.cc:225] Handling call
0507 08:12:14.071423 (+5370992us) ts_tablet_manager.cc:1120] Deleting tablet ec3e1823f9ff4f19be2b13cdcd706f59
0507 08:12:14.071432 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.071525 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5370899}
W20240507 08:12:14.072388 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ec3e1823f9ff4f19be2b13cdcd706f59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ec3e1823f9ff4f19be2b13cdcd706f59 already in progress: creating tablet
I20240507 08:12:14.072793 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 939) took 5391 ms. Trace:
I20240507 08:12:14.073956 5841 rpcz_store.cc:276] 0507 08:12:08.681040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.700528 (+ 19488us) service_pool.cc:225] Handling call
0507 08:12:14.072693 (+5372165us) ts_tablet_manager.cc:1120] Deleting tablet 5b0c793dbc884efd92b49ae7798206a2
0507 08:12:14.072702 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.072777 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5372052}
W20240507 08:12:14.074442 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5b0c793dbc884efd92b49ae7798206a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b0c793dbc884efd92b49ae7798206a2 already in progress: creating tablet
I20240507 08:12:14.074678 7318 tablet.cc:1722] T 835d7b779d724df4863bb13a5140dbf0 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:14.074820 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 944) took 5197 ms. Trace:
I20240507 08:12:14.074887 5841 rpcz_store.cc:276] 0507 08:12:08.877174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.877232 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:14.074750 (+5197518us) ts_tablet_manager.cc:1120] Deleting tablet b4504b1ed3d449619926fe80469d9790
0507 08:12:14.074758 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.074808 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5197309}
W20240507 08:12:14.075209 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b4504b1ed3d449619926fe80469d9790 because the tablet was not found. No further retry: Not found: Tablet not found: b4504b1ed3d449619926fe80469d9790
I20240507 08:12:14.075543 5841 tablet_replica.cc:330] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.076104 7351 tablet.cc:1722] T 835d7b779d724df4863bb13a5140dbf0 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:14.081614 7330 raft_consensus.cc:482] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.081815 7330 raft_consensus.cc:504] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.082448 7330 leader_election.cc:288] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.082717 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b65e251f3d5c4414afe413a54a519d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:14.083402 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.083571 5446 raft_consensus.cc:2454] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.083902 5656 leader_election.cc:302] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.084282 7330 raft_consensus.cc:2781] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.084347 7330 raft_consensus.cc:482] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.084396 7330 raft_consensus.cc:3037] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.084652 5841 raft_consensus.cc:2229] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.084801 5841 raft_consensus.cc:2258] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.085290 5841 ts_tablet_manager.cc:1875] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.085718 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.086735 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58d9a185726144d589e20ce85c4897fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:14.086921 7330 raft_consensus.cc:504] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.087888 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.088114 5446 raft_consensus.cc:3037] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.090584 5840 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.090711 5840 ts_tablet_manager.cc:1367] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.000s
I20240507 08:12:14.090732 5446 raft_consensus.cc:2454] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.091207 5656 leader_election.cc:302] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.091478 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33875331045146a9a02559a0b37d5c6f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.091404 5840 raft_consensus.cc:348] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.091692 5840 raft_consensus.cc:374] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.091750 5840 raft_consensus.cc:729] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.091842 7101 raft_consensus.cc:2781] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.091981 7101 raft_consensus.cc:686] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.091951 5840 consensus_queue.cc:260] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.092213 7101 consensus_queue.cc:237] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.092288 5654 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.092470 5840 ts_tablet_manager.cc:1398] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:14.092722 5840 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:14.092859 5654 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.094137 5840 tablet_bootstrap.cc:654] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.100317 5843 raft_consensus.cc:348] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.101445 7330 leader_election.cc:288] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.102416 5843 raft_consensus.cc:374] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.102525 5843 raft_consensus.cc:729] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.102818 5843 consensus_queue.cc:260] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.102325 5841 ts_tablet_manager.cc:1888] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:14.103204 5841 log.cc:1196] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:12:14.103557 5841 ts_tablet_manager.cc:1909] T a56cb45381604ac9a69e6f3edaa4eefc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.104858 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 943) took 5227 ms. Trace:
I20240507 08:12:14.105175 5841 rpcz_store.cc:276] 0507 08:12:08.877083 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:08.884528 (+ 7445us) service_pool.cc:225] Handling call
0507 08:12:14.075501 (+5190973us) ts_tablet_manager.cc:1120] Deleting tablet a56cb45381604ac9a69e6f3edaa4eefc
0507 08:12:14.075509 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.088235 (+ 12726us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.102225 (+ 13990us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.104838 (+ 2613us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5190803}
I20240507 08:12:14.105942 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet a56cb45381604ac9a69e6f3edaa4eefc (table default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]) successfully deleted
I20240507 08:12:14.104061 5843 ts_tablet_manager.cc:1398] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240507 08:12:14.106484 5843 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.107520 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 949) took 4925 ms. Trace:
I20240507 08:12:14.108644 5841 rpcz_store.cc:276] 0507 08:12:09.182591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.182647 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:14.107399 (+4924752us) ts_tablet_manager.cc:1120] Deleting tablet e7eef600626a4961ab1fae27162537bb
0507 08:12:14.107409 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.107504 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4908783}
W20240507 08:12:14.109319 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e7eef600626a4961ab1fae27162537bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7eef600626a4961ab1fae27162537bb already in progress: creating tablet
I20240507 08:12:14.109711 5841 tablet_replica.cc:330] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.110298 5839 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.108397 5843 tablet_bootstrap.cc:654] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.110364 5841 raft_consensus.cc:2229] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.112713 5841 raft_consensus.cc:2258] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.113391 5841 ts_tablet_manager.cc:1875] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.113940 5839 tablet_bootstrap.cc:654] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.120059 7330 raft_consensus.cc:482] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.120400 7330 raft_consensus.cc:504] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.122633 7330 leader_election.cc:288] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.123255 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.123500 5446 raft_consensus.cc:2454] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.123944 5656 leader_election.cc:302] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.124591 7330 raft_consensus.cc:2781] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.124716 7330 raft_consensus.cc:482] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.124830 7330 raft_consensus.cc:3037] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.125066 7294 raft_consensus.cc:482] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.125181 7294 raft_consensus.cc:504] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.125758 7294 leader_election.cc:288] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.127641 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:14.130407 5377 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.134816 6513 ts_tablet_manager.cc:1888] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.134934 6513 log.cc:1196] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/753019050fcf4dc3b69722c3eb475037
I20240507 08:12:14.135282 6513 ts_tablet_manager.cc:1909] T 753019050fcf4dc3b69722c3eb475037 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.135619 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 988) took 3428 ms. Trace:
I20240507 08:12:14.135710 6513 rpcz_store.cc:276] 0507 08:12:10.706833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.706895 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:14.029467 (+3322572us) ts_tablet_manager.cc:1120] Deleting tablet 753019050fcf4dc3b69722c3eb475037
0507 08:12:14.029477 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.131843 (+102366us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.134727 (+ 2884us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.135602 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3322359}
I20240507 08:12:14.136319 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 753019050fcf4dc3b69722c3eb475037 (table default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]) successfully deleted
I20240507 08:12:14.137324 5840 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.137509 5840 ts_tablet_manager.cc:1367] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20240507 08:12:14.137815 7330 raft_consensus.cc:504] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.138268 5840 raft_consensus.cc:348] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.141242 5840 raft_consensus.cc:374] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.141342 5840 raft_consensus.cc:729] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.141594 5840 consensus_queue.cc:260] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.142467 5840 ts_tablet_manager.cc:1398] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:14.142846 7330 leader_election.cc:288] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.143121 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.143327 5446 raft_consensus.cc:3037] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.146435 5446 raft_consensus.cc:2454] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.146807 5656 leader_election.cc:302] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.147058 7330 raft_consensus.cc:2781] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.147300 7330 raft_consensus.cc:686] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.147603 7330 consensus_queue.cc:237] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.148643 5841 ts_tablet_manager.cc:1888] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:14.148787 5841 log.cc:1196] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/f2de26190bc44916809856d84ad68345
I20240507 08:12:14.149127 5841 ts_tablet_manager.cc:1909] T f2de26190bc44916809856d84ad68345 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.150084 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 951) took 4967 ms. Trace:
I20240507 08:12:14.150993 5841 rpcz_store.cc:276] 0507 08:12:09.182825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.182871 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:14.109666 (+4926795us) ts_tablet_manager.cc:1120] Deleting tablet f2de26190bc44916809856d84ad68345
0507 08:12:14.109675 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.117383 (+ 7708us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.148549 (+ 31166us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.150064 (+ 1515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4911009}
I20240507 08:12:14.151671 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet f2de26190bc44916809856d84ad68345 (table default.Table_f2eabb2165474aadb0528bf86cbc1718 [id=ff9529d9a26e41cb9ebf472b849d6764]) successfully deleted
I20240507 08:12:14.152196 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 952) took 4969 ms. Trace:
I20240507 08:12:14.152479 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 999) took 1823 ms. Trace:
I20240507 08:12:14.152586 5841 rpcz_store.cc:276] 0507 08:12:09.182917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.182961 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:14.152073 (+4969112us) ts_tablet_manager.cc:1120] Deleting tablet d94eb57a25bc440196abe6861ca19b95
0507 08:12:14.152083 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.152180 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4953361}
W20240507 08:12:14.152938 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d94eb57a25bc440196abe6861ca19b95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d94eb57a25bc440196abe6861ca19b95 already in progress: creating tablet
I20240507 08:12:14.153529 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 950) took 4970 ms. Trace:
I20240507 08:12:14.152900 7079 rpcz_store.cc:276] 0507 08:12:12.328565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.328869 (+ 304us) service_pool.cc:225] Handling call
0507 08:12:13.319911 (+991042us) ts_tablet_manager.cc:1120] Deleting tablet 4de7053cfaea4a54a402c9d826613c05
0507 08:12:13.319919 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:13.907567 (+587648us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.059257 (+151690us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.152418 (+ 93161us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":990910}
I20240507 08:12:14.153940 5841 rpcz_store.cc:276] 0507 08:12:09.182719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:09.183125 (+ 406us) service_pool.cc:225] Handling call
0507 08:12:14.153421 (+4970296us) ts_tablet_manager.cc:1120] Deleting tablet bfa2139385d34c37a7ac14d78480a779
0507 08:12:14.153430 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.153509 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4954674}
W20240507 08:12:14.154275 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bfa2139385d34c37a7ac14d78480a779 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfa2139385d34c37a7ac14d78480a779 already in progress: creating tablet
I20240507 08:12:14.154861 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 964) took 4065 ms. Trace:
I20240507 08:12:14.154930 5841 rpcz_store.cc:276] 0507 08:12:10.089373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.091564 (+ 2191us) service_pool.cc:225] Handling call
0507 08:12:14.154760 (+4063196us) ts_tablet_manager.cc:1120] Deleting tablet 1ab94b1a09dd4969b928b017229ce4bf
0507 08:12:14.154772 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.154847 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4062539}
W20240507 08:12:14.155231 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1ab94b1a09dd4969b928b017229ce4bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1ab94b1a09dd4969b928b017229ce4bf already in progress: creating tablet
I20240507 08:12:14.156002 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 963) took 4066 ms. Trace:
I20240507 08:12:14.156342 5841 rpcz_store.cc:276] 0507 08:12:10.089197 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.091824 (+ 2627us) service_pool.cc:225] Handling call
0507 08:12:14.155909 (+4064085us) ts_tablet_manager.cc:1120] Deleting tablet 06898414c5494cd380e0c932f4589014
0507 08:12:14.155917 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.155987 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4062997}
W20240507 08:12:14.156889 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 06898414c5494cd380e0c932f4589014 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06898414c5494cd380e0c932f4589014 already in progress: creating tablet
I20240507 08:12:14.157308 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 965) took 4062 ms. Trace:
I20240507 08:12:14.157630 5841 rpcz_store.cc:276] 0507 08:12:10.094542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.094609 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:14.157218 (+4062609us) ts_tablet_manager.cc:1120] Deleting tablet 2e44c3f6f96e417b92878d8ba4ff88a7
0507 08:12:14.157227 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.157295 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4062248}
I20240507 08:12:14.157730 5255 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
W20240507 08:12:14.158154 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2e44c3f6f96e417b92878d8ba4ff88a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e44c3f6f96e417b92878d8ba4ff88a7 already in progress: creating tablet
I20240507 08:12:14.158695 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 966) took 4037 ms. Trace:
I20240507 08:12:14.159045 5841 rpcz_store.cc:276] 0507 08:12:10.121239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.121414 (+ 175us) service_pool.cc:225] Handling call
0507 08:12:14.158599 (+4037185us) ts_tablet_manager.cc:1120] Deleting tablet bd6ca1a20df748e097de097d1c2d7be3
0507 08:12:14.158608 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.158681 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4036963}
W20240507 08:12:14.159713 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet bd6ca1a20df748e097de097d1c2d7be3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bd6ca1a20df748e097de097d1c2d7be3 already in progress: creating tablet
I20240507 08:12:14.160377 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 967) took 4039 ms. Trace:
I20240507 08:12:14.160645 5841 rpcz_store.cc:276] 0507 08:12:10.121357 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.121666 (+ 309us) service_pool.cc:225] Handling call
0507 08:12:14.159934 (+4038268us) ts_tablet_manager.cc:1120] Deleting tablet d17b6a2f4aa14f2a927fa9ad56a62a5a
0507 08:12:14.160072 (+ 138us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.160357 (+ 285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4038090}
W20240507 08:12:14.161315 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d17b6a2f4aa14f2a927fa9ad56a62a5a already in progress: creating tablet
I20240507 08:12:14.161669 5841 tablet_replica.cc:330] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.162101 5841 raft_consensus.cc:2229] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.162557 5841 raft_consensus.cc:2258] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.163283 5841 ts_tablet_manager.cc:1875] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.166476 6513 tablet_replica.cc:330] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.166733 6513 raft_consensus.cc:2229] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:14.167038 6513 raft_consensus.cc:2258] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.167014 7079 tablet_replica.cc:330] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:14.167322 5840 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.167459 7079 raft_consensus.cc:2229] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.167896 7079 raft_consensus.cc:2258] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.169054 5840 tablet_bootstrap.cc:654] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.170399 7079 ts_tablet_manager.cc:1875] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.173846 6513 ts_tablet_manager.cc:1875] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.183053 5840 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.183298 5840 ts_tablet_manager.cc:1367] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.003s
I20240507 08:12:14.183653 7378 raft_consensus.cc:482] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.183851 7378 raft_consensus.cc:504] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.184051 5840 raft_consensus.cc:348] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.184302 5840 raft_consensus.cc:374] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.184396 5840 raft_consensus.cc:729] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.184655 5840 consensus_queue.cc:260] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.184916 7378 leader_election.cc:288] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.185950 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:14.188760 5840 ts_tablet_manager.cc:1398] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:14.189111 5840 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:14.190414 5377 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.190863 5840 tablet_bootstrap.cc:654] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.193347 6513 ts_tablet_manager.cc:1888] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.193457 6513 log.cc:1196] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c43f3de489904f2ba2a637d408619bfc
I20240507 08:12:14.197075 5841 ts_tablet_manager.cc:1888] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:14.198711 5841 log.cc:1196] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:14.199179 5841 ts_tablet_manager.cc:1909] T aa094773f3bf418492bcd5b5021fd429 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.199662 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.200237 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 981) took 3273 ms. Trace:
I20240507 08:12:14.200320 5841 rpcz_store.cc:276] 0507 08:12:10.926780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.926835 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:14.161627 (+3234792us) ts_tablet_manager.cc:1120] Deleting tablet aa094773f3bf418492bcd5b5021fd429
0507 08:12:14.161636 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.181361 (+ 19725us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.196979 (+ 15618us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.200218 (+ 3239us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3234574}
I20240507 08:12:14.200711 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet aa094773f3bf418492bcd5b5021fd429 (table default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]) successfully deleted
I20240507 08:12:14.201089 5580 raft_consensus.cc:3037] T 371ba277be774d32912d1174bec68b51 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.202543 5841 tablet_replica.cc:330] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.202720 5841 raft_consensus.cc:2229] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.202862 5841 raft_consensus.cc:2258] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.202944 5840 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.203366 5841 ts_tablet_manager.cc:1875] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.203346 5840 ts_tablet_manager.cc:1367] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20240507 08:12:14.203878 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2d913dc032b44a4fbab59b42efcba58c
I20240507 08:12:14.204365 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 2d913dc032b44a4fbab59b42efcba58c deleted and replaced by tablet f1980322500f42b99180b86cc259ca3f
I20240507 08:12:14.205108 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d913dc032b44a4fbab59b42efcba58c with delete_type TABLET_DATA_DELETED (replaced by tablet f1980322500f42b99180b86cc259ca3f) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:14.205236 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124ea88554ec4182a6682457cd58dfe1 consensus state has no leader
I20240507 08:12:14.205770 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 7e26c7ef6f2845cabe09d09045d4feda
I20240507 08:12:14.205796 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.206164 5262 catalog_manager.cc:6554] ReplaceTablet: tablet 7e26c7ef6f2845cabe09d09045d4feda deleted and replaced by tablet fce52cf5871c402caeb769d1ca4c408a
I20240507 08:12:14.206164 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.205600 5840 raft_consensus.cc:348] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.206790 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda with delete_type TABLET_DATA_DELETED (replaced by tablet fce52cf5871c402caeb769d1ca4c408a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.206609 5840 raft_consensus.cc:374] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.206887 5840 raft_consensus.cc:729] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:14.207199 5654 leader_election.cc:341] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.208081 5840 consensus_queue.cc:260] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.208766 5654 leader_election.cc:341] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:14.209460 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0f5ba0221304453a00bb779aee676d4 has no consensus state
I20240507 08:12:14.209517 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d913dc032b44a4fbab59b42efcba58c with delete_type TABLET_DATA_DELETED (replaced by tablet f1980322500f42b99180b86cc259ca3f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.209729 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda with delete_type TABLET_DATA_DELETED (replaced by tablet fce52cf5871c402caeb769d1ca4c408a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.210368 5840 ts_tablet_manager.cc:1398] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240507 08:12:14.211585 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]). Expected version 2 got 0
I20240507 08:12:14.211763 7358 consensus_queue.cc:1035] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.214619 5840 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.217640 6513 ts_tablet_manager.cc:1909] T c43f3de489904f2ba2a637d408619bfc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.218384 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 989) took 3511 ms. Trace:
I20240507 08:12:14.221900 5840 tablet_bootstrap.cc:654] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.237903 7101 raft_consensus.cc:482] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.238147 7101 raft_consensus.cc:504] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.238252 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ba61f9a9c956447589c8e0db1eaa6f6f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.238793 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba61f9a9c956447589c8e0db1eaa6f6f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.238795 7101 leader_election.cc:288] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.239197 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_86003022cb914d9ead81ce912a28d1cd" } modify_external_catalogs: true
I20240507 08:12:14.239224 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.239372 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_86003022cb914d9ead81ce912a28d1cd" } modify_external_catalogs: true
I20240507 08:12:14.239460 5447 raft_consensus.cc:2454] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.240180 5656 leader_election.cc:302] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.241609 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16b1817c74e5443c9c83a953e4514b13" } modify_external_catalogs: true
I20240507 08:12:14.241760 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_16b1817c74e5443c9c83a953e4514b13" } modify_external_catalogs: true
I20240507 08:12:14.242898 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_091c1a53ece34eb3bbc8265fa45f9088"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.243098 7101 raft_consensus.cc:2781] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240507 08:12:14.243407 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_091c1a53ece34eb3bbc8265fa45f9088 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.243383 7101 raft_consensus.cc:482] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.243701 7101 raft_consensus.cc:3037] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.244509 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7d3cd26b01594f059ae92b67fe87d846"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.244952 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d3cd26b01594f059ae92b67fe87d846 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.245949 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a36aab1e10b41c589b8b0eac87af0db" } modify_external_catalogs: true
I20240507 08:12:14.246101 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8a36aab1e10b41c589b8b0eac87af0db" } modify_external_catalogs: true
I20240507 08:12:14.247313 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:14.247918 5654 leader_election.cc:341] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.249037 7101 raft_consensus.cc:504] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.252826 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.252832 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcefbe3a60f84cadb5fd3e69c6efdd02" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.253021 5447 raft_consensus.cc:3037] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:14.255399 5654 leader_election.cc:341] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.256043 5447 raft_consensus.cc:2454] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.256028 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.256492 5656 leader_election.cc:302] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.256717 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4de7053cfaea4a54a402c9d826613c05 (table default.Table_c076159e489f4d5a97e67533975f0a43 [id=55e5bea6222c4537813952f6fdc7387e]) successfully deleted
I20240507 08:12:14.256867 7100 raft_consensus.cc:2781] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.257001 7100 raft_consensus.cc:686] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:14.257202 5378 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.257302 5378 leader_election.cc:302] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.257318 7100 consensus_queue.cc:237] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.257786 5378 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.257880 5378 leader_election.cc:302] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.257892 7378 raft_consensus.cc:2726] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.258083 7378 raft_consensus.cc:2726] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.259043 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e26c7ef6f2845cabe09d09045d4feda with delete_type TABLET_DATA_DELETED (replaced by tablet fce52cf5871c402caeb769d1ca4c408a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.262871 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 11e779c5c97c44e5bcc7c0eb0194fd95
I20240507 08:12:14.264199 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.264312 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.268030 7101 leader_election.cc:288] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.276441 5843 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.276698 5843 ts_tablet_manager.cc:1367] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.170s user 0.013s sys 0.013s
I20240507 08:12:14.277779 7101 raft_consensus.cc:482] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.278012 7101 raft_consensus.cc:504] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.279492 7101 leader_election.cc:288] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.279790 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.279899 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.280051 5446 raft_consensus.cc:2454] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.280472 5656 leader_election.cc:302] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.281605 7101 raft_consensus.cc:2781] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.281764 7101 raft_consensus.cc:482] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.277477 5843 raft_consensus.cc:348] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.285013 5843 raft_consensus.cc:374] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.287456 5843 raft_consensus.cc:729] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.287806 5843 consensus_queue.cc:260] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.292415 5843 ts_tablet_manager.cc:1398] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240507 08:12:14.292809 7330 raft_consensus.cc:482] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.294240 5843 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.298365 5843 tablet_bootstrap.cc:654] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:14.301867 5654 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.301898 7101 raft_consensus.cc:3037] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.303400 7330 raft_consensus.cc:504] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.304553 7330 leader_election.cc:288] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.304762 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d913dc032b44a4fbab59b42efcba58c with delete_type TABLET_DATA_DELETED (replaced by tablet f1980322500f42b99180b86cc259ca3f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.305119 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.305375 5446 raft_consensus.cc:2454] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.306744 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.307987 5656 leader_election.cc:302] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.308544 7330 raft_consensus.cc:2781] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.308691 7330 raft_consensus.cc:482] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.308938 7330 raft_consensus.cc:3037] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.310631 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:13 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.311803 7330 raft_consensus.cc:504] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.312039 5654 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.316139 7101 raft_consensus.cc:504] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.317133 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.317494 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "767905763ba14cb8bbc52151ae9f17e6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.317807 5446 raft_consensus.cc:3037] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.318420 7391 tablet.cc:1722] T 835d7b779d724df4863bb13a5140dbf0 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:14.321316 5446 raft_consensus.cc:2454] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.321805 5656 leader_election.cc:302] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.322240 7333 raft_consensus.cc:2781] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.322439 7333 raft_consensus.cc:686] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.322768 7333 consensus_queue.cc:237] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.324044 6784 raft_consensus.cc:482] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:14.324141 5654 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.324187 6784 raft_consensus.cc:504] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.324810 6784 leader_election.cc:288] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.325685 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.325882 5447 raft_consensus.cc:2454] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.326345 5656 leader_election.cc:302] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.290112 5841 ts_tablet_manager.cc:1888] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.326529 5841 log.cc:1196] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d6f8f4205f1f48abaf4fcd7113ad44a8
I20240507 08:12:14.326592 6675 raft_consensus.cc:2781] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.326707 6675 raft_consensus.cc:482] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.326810 6675 raft_consensus.cc:3037] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.327226 5841 ts_tablet_manager.cc:1909] T d6f8f4205f1f48abaf4fcd7113ad44a8 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.327598 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 364903504aa14d08b050a0cf83b82c71
I20240507 08:12:14.328619 5264 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24] alter complete (version 1)
I20240507 08:12:14.328802 7330 leader_election.cc:288] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.330241 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 364903504aa14d08b050a0cf83b82c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.330268 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 364903504aa14d08b050a0cf83b82c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.331526 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.331704 5447 raft_consensus.cc:3037] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.332724 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.332886 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.335108 5447 raft_consensus.cc:2454] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.335494 6675 raft_consensus.cc:504] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.335803 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2b0061a7e41c4d66aaff6de6bad0e892" } new_table_name: "default.Table_f7a84b84863d462192343bcb92bd888d" modify_external_catalogs: true
I20240507 08:12:14.336275 6675 leader_election.cc:288] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.336994 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85dc58133d2b4f5496b4e570ae6494c0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.337296 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eb491699584f4d18ad9cb26ed87fd99b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.337775 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 364903504aa14d08b050a0cf83b82c71 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:14.337889 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb491699584f4d18ad9cb26ed87fd99b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.339205 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bc1f5c0697f04fbba453282facb1a6a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.339612 5656 leader_election.cc:302] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:14.339712 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc1f5c0697f04fbba453282facb1a6a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.339818 5840 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.340484 6675 raft_consensus.cc:2781] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.340656 6675 raft_consensus.cc:686] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.340840 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5c436dbf3c0f4f879e1d3a57d6405594"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.340951 6675 consensus_queue.cc:237] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.341261 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c436dbf3c0f4f879e1d3a57d6405594 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.341926 5840 ts_tablet_manager.cc:1367] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.126s user 0.026s sys 0.026s
I20240507 08:12:14.342233 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.342429 5447 raft_consensus.cc:3037] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.342896 5840 raft_consensus.cc:348] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.343147 5840 raft_consensus.cc:374] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:14.343372 5654 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.343529 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.343894 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.343992 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.344035 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 753019050fcf4dc3b69722c3eb475037 with delete_type TABLET_DATA_DELETED (replaced by tablet aca8c38841db45e38ae5d8d965a16d0e) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.344833 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.345208 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.345113 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b9e715a29b0643b6ac61f9e73874c25c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.345386 5447 raft_consensus.cc:2454] T b9e715a29b0643b6ac61f9e73874c25c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:14.345434 5654 leader_election.cc:341] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.343240 5840 raft_consensus.cc:729] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.345849 5656 leader_election.cc:302] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.345949 7101 leader_election.cc:288] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.345909 5840 consensus_queue.cc:260] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.345687 5654 leader_election.cc:341] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.346803 6675 raft_consensus.cc:482] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.347393 6675 raft_consensus.cc:504] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.348292 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 982) took 3420 ms. Trace:
I20240507 08:12:14.348377 5841 rpcz_store.cc:276] 0507 08:12:10.927739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.927790 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:14.202484 (+3274694us) ts_tablet_manager.cc:1120] Deleting tablet d6f8f4205f1f48abaf4fcd7113ad44a8
0507 08:12:14.202496 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.232309 (+ 29813us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.289988 (+ 57679us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.348273 (+ 58285us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3274526}
I20240507 08:12:14.348912 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d6f8f4205f1f48abaf4fcd7113ad44a8 (table default.Table_87bb488658cd4819a51ff1ff0993ceab [id=799497aabd9b4f17bc42b655e947b3ed]) successfully deleted
I20240507 08:12:14.349231 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 983) took 3421 ms. Trace:
I20240507 08:12:14.349224 6675 leader_election.cc:288] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.349313 5841 rpcz_store.cc:276] 0507 08:12:10.928232 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.928278 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:14.348639 (+3420361us) ts_tablet_manager.cc:1120] Deleting tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd
0507 08:12:14.349132 (+ 493us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.349214 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3420215}
W20240507 08:12:14.350545 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 302fbf120e9d472ba6bdf9e6cbdf6dbd because the tablet was not found. No further retry: Not found: Tablet not found: 302fbf120e9d472ba6bdf9e6cbdf6dbd
I20240507 08:12:14.347004 5840 ts_tablet_manager.cc:1398] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:14.351840 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b0542325c80241fca7e317cda64302a9
I20240507 08:12:14.352728 5839 tablet_bootstrap.cc:492] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.352907 5839 ts_tablet_manager.cc:1367] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.243s user 0.004s sys 0.003s
I20240507 08:12:14.353011 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.353205 5446 raft_consensus.cc:2454] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.353451 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet b0542325c80241fca7e317cda64302a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.353746 5839 raft_consensus.cc:348] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.353947 5839 raft_consensus.cc:374] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.354014 5839 raft_consensus.cc:729] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.354532 5839 consensus_queue.cc:260] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.355072 5839 ts_tablet_manager.cc:1398] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:14.355333 5839 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.356721 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet b0542325c80241fca7e317cda64302a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.358042 5839 tablet_bootstrap.cc:654] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.358891 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.351342 5840 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.346194 7101 raft_consensus.cc:2781] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.356701 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet b0542325c80241fca7e317cda64302a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:14.357194 5654 consensus_peers.cc:485] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.359612 7101 raft_consensus.cc:686] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:14.360855 5654 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.361194 5840 tablet_bootstrap.cc:654] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.360746 7101 consensus_queue.cc:237] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.366003 5841 tablet_replica.cc:330] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.366287 5841 raft_consensus.cc:2229] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.366495 5841 raft_consensus.cc:2258] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.367350 5841 ts_tablet_manager.cc:1875] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.372030 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8cf67af06cf44e458ec7c70c6a18ffd0" } new_table_name: "default.Table_aef25ed0324c427ea943cf7fa3bd03d5" modify_external_catalogs: true
I20240507 08:12:14.374717 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.376091 5841 ts_tablet_manager.cc:1888] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.376272 5841 log.cc:1196] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:12:14.376564 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 79524df2415146ab926425c42068626c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:04 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.377411 5841 ts_tablet_manager.cc:1909] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.377991 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:14.381485 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ab350a3f4f3341c2ad0dda3ce054e199
I20240507 08:12:14.381666 5584 raft_consensus.cc:3037] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.382978 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.383467 5656 leader_election.cc:302] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.383951 7333 raft_consensus.cc:2781] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.384125 7333 raft_consensus.cc:482] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.384341 7333 raft_consensus.cc:3037] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.384698 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.386864 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet ab350a3f4f3341c2ad0dda3ce054e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.387035 7333 raft_consensus.cc:504] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.388029 7333 leader_election.cc:288] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:14.388698 5656 consensus_peers.cc:485] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a56cb45381604ac9a69e6f3edaa4eefc. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.388676 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_055ab044460f457787d9b3e18fea3729" } new_table_name: "default.Table_702bfc8186ae44c284e968ed2b28120a" modify_external_catalogs: true
W20240507 08:12:14.389122 5656 consensus_peers.cc:485] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a02a5a493bc47cdbb6f7bd5b5c28e7f. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.390702 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.390877 5446 raft_consensus.cc:3037] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.391330 7201 log.cc:824] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Log is configured to *not* fsync() on all Append() calls
I20240507 08:12:14.393412 5446 raft_consensus.cc:2454] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.394615 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 984) took 3465 ms. Trace:
I20240507 08:12:14.394680 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1fadbfd330b4e699e151f83125da281" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.394881 5841 rpcz_store.cc:276] 0507 08:12:10.928711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.928755 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:14.350251 (+3421496us) ts_tablet_manager.cc:1120] Deleting tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f
0507 08:12:14.365946 (+ 15695us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.370621 (+ 4675us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.375936 (+ 5315us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.394596 (+ 18660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3421354}
I20240507 08:12:14.395510 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (table default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]) successfully deleted
I20240507 08:12:14.395578 5841 tablet_replica.cc:330] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.396309 5841 raft_consensus.cc:2229] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.396602 5841 raft_consensus.cc:2258] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.397447 5841 ts_tablet_manager.cc:1875] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.399510 7079 ts_tablet_manager.cc:1888] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.399681 7079 log.cc:1196] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c43f3de489904f2ba2a637d408619bfc
I20240507 08:12:14.400096 7079 ts_tablet_manager.cc:1909] T c43f3de489904f2ba2a637d408619bfc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:14.400442 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1000) took 2003 ms. Trace:
I20240507 08:12:14.400566 7079 rpcz_store.cc:276] 0507 08:12:12.397139 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.397197 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:14.166873 (+1769676us) ts_tablet_manager.cc:1120] Deleting tablet c43f3de489904f2ba2a637d408619bfc
0507 08:12:14.166927 (+ 54us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.311893 (+144966us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.399417 (+ 87524us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.400424 (+ 1007us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1769428}
I20240507 08:12:14.401223 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c43f3de489904f2ba2a637d408619bfc (table default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]) successfully deleted
I20240507 08:12:14.218501 6513 rpcz_store.cc:276] 0507 08:12:10.706936 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707007 (+ 71us) service_pool.cc:225] Handling call
0507 08:12:14.135969 (+3428962us) ts_tablet_manager.cc:1120] Deleting tablet c43f3de489904f2ba2a637d408619bfc
0507 08:12:14.135977 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.190261 (+ 54284us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.193266 (+ 3005us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.218353 (+ 25087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428827}
I20240507 08:12:14.403218 5656 leader_election.cc:302] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.407876 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1f3f2ed28af24116b1d81eecee74897c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.408248 5841 ts_tablet_manager.cc:1888] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.408412 5841 log.cc:1196] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/6b4c69d38e484f5c819b7dd915bf4410
W20240507 08:12:14.408540 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f3f2ed28af24116b1d81eecee74897c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.409437 5841 ts_tablet_manager.cc:1909] T 6b4c69d38e484f5c819b7dd915bf4410 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.410661 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 985) took 3480 ms. Trace:
I20240507 08:12:14.410802 5841 rpcz_store.cc:276] 0507 08:12:10.930316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.930358 (+ 42us) service_pool.cc:225] Handling call
0507 08:12:14.395508 (+3465150us) ts_tablet_manager.cc:1120] Deleting tablet 6b4c69d38e484f5c819b7dd915bf4410
0507 08:12:14.395518 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.401939 (+ 6421us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.408053 (+ 6114us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.410620 (+ 2567us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3465013}
I20240507 08:12:14.411423 5841 tablet_replica.cc:330] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.411703 5841 raft_consensus.cc:2229] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.411993 5841 raft_consensus.cc:2258] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.413304 5841 ts_tablet_manager.cc:1875] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.414629 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 6b4c69d38e484f5c819b7dd915bf4410 (table default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]) successfully deleted
I20240507 08:12:14.417848 7394 raft_consensus.cc:2781] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.418040 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c43f3de489904f2ba2a637d408619bfc (table default.Table_ee9c3ec6f6cf4eaf9977830d2c43c4ef [id=0895032e26b2485fb0a2c6fe025bd04e]) successfully deleted
I20240507 08:12:14.418061 7394 raft_consensus.cc:686] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.419263 7394 consensus_queue.cc:237] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.424198 5654 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.430328 5839 tablet_bootstrap.cc:492] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.430544 5839 ts_tablet_manager.cc:1367] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.000s
I20240507 08:12:14.431290 5839 raft_consensus.cc:348] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.431547 5839 raft_consensus.cc:374] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.431646 5839 raft_consensus.cc:729] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.431910 5839 consensus_queue.cc:260] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.432401 5839 ts_tablet_manager.cc:1398] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:14.432683 5839 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.433822 6513 tablet_replica.cc:330] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.434383 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2915a6cc4171446abe445f6c4fb1eb44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.435104 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2915a6cc4171446abe445f6c4fb1eb44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.441309 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 55bacb9fd24b46709704eb852e051eb1
I20240507 08:12:14.442462 5841 ts_tablet_manager.cc:1888] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.442958 5841 log.cc:1196] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/049483b0533642f2b46d8fa0e9b123cc
I20240507 08:12:14.444873 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bacb9fd24b46709704eb852e051eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.445977 6513 raft_consensus.cc:2229] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:14.446082 7329 raft_consensus.cc:482] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.446214 7329 raft_consensus.cc:504] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.446987 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bacb9fd24b46709704eb852e051eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.447295 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 55bacb9fd24b46709704eb852e051eb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:14.447610 5517 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 753019050fcf4dc3b69722c3eb475037
I20240507 08:12:14.448863 5841 ts_tablet_manager.cc:1909] T 049483b0533642f2b46d8fa0e9b123cc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.450556 6513 raft_consensus.cc:2258] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.453692 6513 ts_tablet_manager.cc:1875] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.455018 5840 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.456943 5840 ts_tablet_manager.cc:1367] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.011s
I20240507 08:12:14.401564 7079 tablet_replica.cc:330] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:14.461691 5840 raft_consensus.cc:348] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.462908 5840 raft_consensus.cc:374] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.463402 5843 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.463614 5843 ts_tablet_manager.cc:1367] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.169s user 0.003s sys 0.000s
I20240507 08:12:14.463586 5840 raft_consensus.cc:729] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.465284 5843 raft_consensus.cc:348] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.464740 5840 consensus_queue.cc:260] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.465675 5843 raft_consensus.cc:374] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.465801 5843 raft_consensus.cc:729] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.466126 5843 consensus_queue.cc:260] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.466845 5843 ts_tablet_manager.cc:1398] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:14.468086 7329 leader_election.cc:288] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:14.468467 5654 consensus_peers.cc:485] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:14.469022 5520 leader_election.cc:341] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 753019050fcf4dc3b69722c3eb475037
I20240507 08:12:14.469141 5520 leader_election.cc:302] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:14.469499 7329 raft_consensus.cc:2726] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.470158 7378 raft_consensus.cc:482] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.470285 7378 raft_consensus.cc:504] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.470849 7378 leader_election.cc:288] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.471185 5840 ts_tablet_manager.cc:1398] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240507 08:12:14.471462 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.471706 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet 19b77b6ef778454e91105e1e9b60020d from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:14.472034 5378 leader_election.cc:341] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.472594 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:14.472975 5377 leader_election.cc:341] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.473069 5377 leader_election.cc:302] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.473474 7378 raft_consensus.cc:2726] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.474272 7079 raft_consensus.cc:2229] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.474463 7079 raft_consensus.cc:2258] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.475036 7079 ts_tablet_manager.cc:1875] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.477118 5840 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.486395 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet 62d6eac6baa249eb847c5625d64f998f from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:14.488773 7333 consensus_queue.cc:1035] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.477144 5839 tablet_bootstrap.cc:654] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.489553 5840 tablet_bootstrap.cc:654] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.502321 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 986) took 3571 ms. Trace:
I20240507 08:12:14.502708 5841 rpcz_store.cc:276] 0507 08:12:10.930745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.930786 (+ 41us) service_pool.cc:225] Handling call
0507 08:12:14.411353 (+3480567us) ts_tablet_manager.cc:1120] Deleting tablet 049483b0533642f2b46d8fa0e9b123cc
0507 08:12:14.411385 (+ 32us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.419982 (+ 8597us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.442056 (+ 22074us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.502297 (+ 60241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3480428}
I20240507 08:12:14.503436 5841 tablet_replica.cc:330] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.503396 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 049483b0533642f2b46d8fa0e9b123cc (table default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]) successfully deleted
I20240507 08:12:14.503683 5841 raft_consensus.cc:2229] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.503955 5841 raft_consensus.cc:2258] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.504776 5841 ts_tablet_manager.cc:1875] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.509460 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c4186447e2f943c5b1eedcb89625e756" } new_table_name: "default.Table_e77fae0cabcb447b9e8e6777828e20cf" modify_external_catalogs: true
I20240507 08:12:14.505398 5843 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.513696 5843 tablet_bootstrap.cc:654] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.516511 5840 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
W20240507 08:12:14.517427 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a40a4e36cc6145cf8311e06720cbd4c2 consensus state has no leader
W20240507 08:12:14.520236 5656 consensus_peers.cc:485] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6b4c69d38e484f5c819b7dd915bf4410. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.521553 5840 ts_tablet_manager.cc:1367] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.004s
I20240507 08:12:14.522385 5840 raft_consensus.cc:348] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.523198 5840 raft_consensus.cc:374] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.523313 5840 raft_consensus.cc:729] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.523624 5840 consensus_queue.cc:260] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.524158 5840 ts_tablet_manager.cc:1398] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:14.527006 5840 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.530783 6513 ts_tablet_manager.cc:1888] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.531549 6513 log.cc:1196] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/44ec0526e1914221bc22b3e75ea01e0d
I20240507 08:12:14.533643 6513 ts_tablet_manager.cc:1909] T 44ec0526e1914221bc22b3e75ea01e0d P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:14.537635 5654 consensus_peers.cc:485] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.538517 7079 ts_tablet_manager.cc:1888] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:12:14.538691 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 225565us
I20240507 08:12:14.540141 5840 tablet_bootstrap.cc:654] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.547279 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.549837 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.553406 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 991) took 3846 ms. Trace:
I20240507 08:12:14.553819 6513 rpcz_store.cc:276] 0507 08:12:10.707116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707155 (+ 39us) service_pool.cc:225] Handling call
0507 08:12:14.401945 (+3694790us) ts_tablet_manager.cc:1120] Deleting tablet 44ec0526e1914221bc22b3e75ea01e0d
0507 08:12:14.401955 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.476939 (+ 74984us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.530657 (+ 53718us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.553335 (+ 22678us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694631}
I20240507 08:12:14.554978 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 44ec0526e1914221bc22b3e75ea01e0d (table default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]) successfully deleted
I20240507 08:12:14.555193 6513 tablet_replica.cc:330] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.556789 6513 raft_consensus.cc:2229] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.557981 6513 raft_consensus.cc:2258] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:14.564514 5654 consensus_peers.cc:485] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.568780 6513 ts_tablet_manager.cc:1875] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:14.569747 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c11eae1778e64e09a1bd0dcf644dfc39 consensus state has no leader
I20240507 08:12:14.573515 7351 tablet.cc:1722] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:14.574375 7318 tablet.cc:1722] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:14.576195 5841 ts_tablet_manager.cc:1888] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.576370 5841 log.cc:1196] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b2df872d04b94c30872b03367df3cc31
I20240507 08:12:14.577386 5841 ts_tablet_manager.cc:1909] T b2df872d04b94c30872b03367df3cc31 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.578603 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 987) took 3635 ms. Trace:
I20240507 08:12:14.578773 5841 rpcz_store.cc:276] 0507 08:12:10.942837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.942882 (+ 45us) service_pool.cc:225] Handling call
0507 08:12:14.503388 (+3560506us) ts_tablet_manager.cc:1120] Deleting tablet b2df872d04b94c30872b03367df3cc31
0507 08:12:14.503397 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.508838 (+ 5441us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.576090 (+ 67252us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.578585 (+ 2495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3560338}
I20240507 08:12:14.579607 5841 tablet_replica.cc:330] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.580154 5841 raft_consensus.cc:2229] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:14.580715 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b2df872d04b94c30872b03367df3cc31 (table default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]) successfully deleted
I20240507 08:12:14.581722 5840 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.581908 5840 ts_tablet_manager.cc:1367] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.006s
I20240507 08:12:14.582639 5840 raft_consensus.cc:348] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.582882 5840 raft_consensus.cc:374] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.582973 5840 raft_consensus.cc:729] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.583206 5840 consensus_queue.cc:260] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.583840 5840 ts_tablet_manager.cc:1398] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:14.584668 5840 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.585984 5840 tablet_bootstrap.cc:654] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.593071 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ca9f79f6fd724517a82be3b0bf66dcf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.593830 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca9f79f6fd724517a82be3b0bf66dcf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.595108 7394 raft_consensus.cc:482] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.595235 7394 raft_consensus.cc:504] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.595871 7394 leader_election.cc:288] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:14.596854 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.597035 5446 raft_consensus.cc:2454] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.597492 5656 leader_election.cc:302] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.597914 7394 raft_consensus.cc:2781] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.597985 7394 raft_consensus.cc:482] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.598039 7394 raft_consensus.cc:3037] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.600461 7394 raft_consensus.cc:504] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:14.603206 5656 consensus_peers.cc:485] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d617213b959452ab6519809e56db128. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.604066 7394 leader_election.cc:288] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:14.604331 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.604497 5446 raft_consensus.cc:3037] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.580773 5841 raft_consensus.cc:2258] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.607028 5446 raft_consensus.cc:2454] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.607523 5656 leader_election.cc:302] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.608013 7333 raft_consensus.cc:2781] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.608117 7333 raft_consensus.cc:686] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.608342 7333 consensus_queue.cc:237] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.609336 5841 ts_tablet_manager.cc:1875] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.612243 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:14.628726 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_97fdb4f0eb6f4975882ec9a27516eb43" } new_table_name: "default.Table_8d1d33ea89b74c3da59e90ff6253f4e0" modify_external_catalogs: true
W20240507 08:12:14.630208 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80bfaa8d097646e3bcce7022f314f4fa consensus state has no leader
I20240507 08:12:14.630538 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35152415e8264b5495c852cb2a68506c" } modify_external_catalogs: true
I20240507 08:12:14.630695 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35152415e8264b5495c852cb2a68506c" } modify_external_catalogs: true
I20240507 08:12:14.631408 5840 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.631543 5840 ts_tablet_manager.cc:1367] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
I20240507 08:12:14.635737 7394 consensus_queue.cc:1035] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.636237 6513 ts_tablet_manager.cc:1888] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.640033 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_82aad3e9a8064c75a66f0675c60c76e1" } modify_external_catalogs: true
I20240507 08:12:14.640237 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_82aad3e9a8064c75a66f0675c60c76e1" } modify_external_catalogs: true
I20240507 08:12:14.632223 5840 raft_consensus.cc:348] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.641013 5840 raft_consensus.cc:374] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.641206 5840 raft_consensus.cc:729] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.641499 6513 log.cc:1196] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/565ac6091f5e48b1b024094d5560e275
I20240507 08:12:14.641535 5840 consensus_queue.cc:260] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.642971 5840 ts_tablet_manager.cc:1398] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20240507 08:12:14.644392 6513 ts_tablet_manager.cc:1909] T 565ac6091f5e48b1b024094d5560e275 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.645401 5840 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.648157 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 990) took 3940 ms. Trace:
I20240507 08:12:14.651122 7101 consensus_queue.cc:1035] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.651782 5840 tablet_bootstrap.cc:654] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.654554 5843 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.654760 5843 ts_tablet_manager.cc:1367] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.149s user 0.012s sys 0.022s
I20240507 08:12:14.648993 6513 rpcz_store.cc:276] 0507 08:12:10.707020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707172 (+ 152us) service_pool.cc:225] Handling call
0507 08:12:14.555103 (+3847931us) ts_tablet_manager.cc:1120] Deleting tablet 565ac6091f5e48b1b024094d5560e275
0507 08:12:14.555112 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.605331 (+ 50219us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.635246 (+ 29915us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.647840 (+ 12594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3847695}
I20240507 08:12:14.655582 5843 raft_consensus.cc:348] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.655848 5843 raft_consensus.cc:374] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.655896 6513 tablet_replica.cc:330] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.655952 5843 raft_consensus.cc:729] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.656193 5843 consensus_queue.cc:260] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.656754 5843 ts_tablet_manager.cc:1398] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:14.657066 5843 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.656415 6513 raft_consensus.cc:2229] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:14.658782 5843 tablet_bootstrap.cc:654] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.659719 6513 raft_consensus.cc:2258] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:14.661365 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 (table default.Table_964680dde8ab415a8553e6ac5bdb6a72 [id=35d988e8eed14dad8783ccafffeb562f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet ace1fdcf3dd8423f9bfd2c77fdc8a939 consensus state has no leader
I20240507 08:12:14.603924 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.604393 5839 tablet_bootstrap.cc:492] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.663297 5839 ts_tablet_manager.cc:1367] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.231s user 0.000s sys 0.003s
I20240507 08:12:14.610469 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f01e3311a8b4553a31914922fb08d13" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:14.663858 5654 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:14.664170 5654 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.664073 5839 raft_consensus.cc:348] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.664269 5839 raft_consensus.cc:374] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.664338 5839 raft_consensus.cc:729] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.664676 5839 consensus_queue.cc:260] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.665150 6513 ts_tablet_manager.cc:1875] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.665619 5839 ts_tablet_manager.cc:1398] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:14.665951 7079 log.cc:1196] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/3b5e3d23bbb6439a87b303dff1b44a1d
I20240507 08:12:14.666509 7079 ts_tablet_manager.cc:1909] T 3b5e3d23bbb6439a87b303dff1b44a1d P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:14.667490 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c35bc706b22451f90611a3e7bc81f4d" } modify_external_catalogs: true
I20240507 08:12:14.667704 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1c35bc706b22451f90611a3e7bc81f4d" } modify_external_catalogs: true
I20240507 08:12:14.668741 5839 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.670495 5839 tablet_bootstrap.cc:654] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.672822 7101 raft_consensus.cc:482] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.672959 7101 raft_consensus.cc:504] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.673262 5841 ts_tablet_manager.cc:1888] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.673367 5841 log.cc:1196] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/565ac6091f5e48b1b024094d5560e275
I20240507 08:12:14.673616 7101 leader_election.cc:288] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.675674 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.675933 5446 raft_consensus.cc:2454] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:14.676106 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 453618a1fb9b415ca06a4567c63d4a44 has no consensus state
I20240507 08:12:14.676355 5656 leader_election.cc:302] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.676661 7333 raft_consensus.cc:2781] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.676779 7333 raft_consensus.cc:482] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.676620 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_42a34eaa8a514f349f1a092abfff155f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.676990 7333 raft_consensus.cc:3037] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.677754 5841 ts_tablet_manager.cc:1909] T 565ac6091f5e48b1b024094d5560e275 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.679455 7333 raft_consensus.cc:504] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.679960 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:14.680768 5654 leader_election.cc:341] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.681988 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.682794 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53bc9bf1c04e4015a152cb750184c9ce" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:14.682965 5437 raft_consensus.cc:3037] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.684006 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1001) took 2193 ms. Trace:
I20240507 08:12:14.684185 7079 rpcz_store.cc:276] 0507 08:12:12.490123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.490173 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:14.401519 (+1911346us) ts_tablet_manager.cc:1120] Deleting tablet 3b5e3d23bbb6439a87b303dff1b44a1d
0507 08:12:14.401528 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.534748 (+133220us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.538427 (+ 3679us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.683989 (+145562us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12662,"tablet-delete.queue_time_us":1911150}
I20240507 08:12:14.684950 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1007) took 2116 ms. Trace:
I20240507 08:12:14.685083 7079 rpcz_store.cc:276] 0507 08:12:12.568414 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.568461 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:14.684750 (+2116289us) ts_tablet_manager.cc:1120] Deleting tablet 1578a63b4e3e4743b1057fa69653cbef
0507 08:12:14.684759 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.684906 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116115}
I20240507 08:12:14.685380 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 3b5e3d23bbb6439a87b303dff1b44a1d (table default.Table_efefbeaaab494b34a0acd79bec1f4185 [id=2a32838c39744c84b5cfc5792d5893e7]) successfully deleted
I20240507 08:12:14.685771 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1006) took 2117 ms. Trace:
I20240507 08:12:14.685931 7079 rpcz_store.cc:276] 0507 08:12:12.568308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.568765 (+ 457us) service_pool.cc:225] Handling call
0507 08:12:14.685523 (+2116758us) ts_tablet_manager.cc:1120] Deleting tablet 2b14c80b96bb43088e1d1354a1dbdd31
0507 08:12:14.685531 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.685724 (+ 193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2116601}
W20240507 08:12:14.686120 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1578a63b4e3e4743b1057fa69653cbef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1578a63b4e3e4743b1057fa69653cbef already in progress: creating tablet
W20240507 08:12:14.686255 5654 leader_election.cc:341] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.686538 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1011) took 2025 ms. Trace:
W20240507 08:12:14.686704 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2b14c80b96bb43088e1d1354a1dbdd31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b14c80b96bb43088e1d1354a1dbdd31 already in progress: creating tablet
W20240507 08:12:14.687125 5656 consensus_peers.cc:485] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 049483b0533642f2b46d8fa0e9b123cc. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.686537 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 988) took 3742 ms. Trace:
I20240507 08:12:14.686756 7079 rpcz_store.cc:276] 0507 08:12:12.661422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.661495 (+ 73us) service_pool.cc:225] Handling call
0507 08:12:14.686378 (+2024883us) ts_tablet_manager.cc:1120] Deleting tablet 5855bc973d354a69a0579d9a76b3f92b
0507 08:12:14.686388 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.686525 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2024650}
I20240507 08:12:14.688293 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1014) took 1973 ms. Trace:
I20240507 08:12:14.688478 7079 rpcz_store.cc:276] 0507 08:12:12.715169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.717073 (+ 1904us) service_pool.cc:225] Handling call
0507 08:12:14.688094 (+1971021us) ts_tablet_manager.cc:1120] Deleting tablet 37a23d112fcd4ea1b0c2f86133e82e32
0507 08:12:14.688103 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.688248 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1869245}
I20240507 08:12:14.685541 5437 raft_consensus.cc:2454] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:14.689267 5656 leader_election.cc:302] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.689540 7333 leader_election.cc:288] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.689519 7101 raft_consensus.cc:2781] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:14.689718 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5855bc973d354a69a0579d9a76b3f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5855bc973d354a69a0579d9a76b3f92b already in progress: creating tablet
I20240507 08:12:14.689749 7101 raft_consensus.cc:686] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
W20240507 08:12:14.689934 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 37a23d112fcd4ea1b0c2f86133e82e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a23d112fcd4ea1b0c2f86133e82e32 already in progress: creating tablet
I20240507 08:12:14.689972 7101 consensus_queue.cc:237] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.689667 5841 rpcz_store.cc:276] 0507 08:12:10.943425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.943469 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:14.579518 (+3636049us) ts_tablet_manager.cc:1120] Deleting tablet 565ac6091f5e48b1b024094d5560e275
0507 08:12:14.579528 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.637932 (+ 58404us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.673151 (+ 35219us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.686211 (+ 13060us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635905}
I20240507 08:12:14.694474 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 989) took 3750 ms. Trace:
I20240507 08:12:14.695866 7329 raft_consensus.cc:482] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.696010 7329 raft_consensus.cc:504] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.696596 7329 leader_election.cc:288] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.695025 5841 rpcz_store.cc:276] 0507 08:12:10.944004 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.944047 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:14.693502 (+3749455us) ts_tablet_manager.cc:1120] Deleting tablet 9ccbe2e368d14b56859d62d1cc742f90
0507 08:12:14.693655 (+ 153us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.694220 (+ 565us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3749182}
I20240507 08:12:14.697366 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:14.697549 5718 raft_consensus.cc:2454] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 0.
W20240507 08:12:14.697851 5520 leader_election.cc:341] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 79524df2415146ab926425c42068626c
I20240507 08:12:14.698189 5517 leader_election.cc:302] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters: dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:14.698552 7329 raft_consensus.cc:2781] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.698632 7329 raft_consensus.cc:482] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.698694 7329 raft_consensus.cc:3037] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.700922 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 999) took 3669 ms. Trace:
I20240507 08:12:14.701117 7329 raft_consensus.cc:504] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.701303 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4306751d16444ec5948c2ba5831063f4
I20240507 08:12:14.702064 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79524df2415146ab926425c42068626c" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:14.702242 5718 raft_consensus.cc:3037] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.701689 5841 rpcz_store.cc:276] 0507 08:12:11.031075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.031149 (+ 74us) service_pool.cc:225] Handling call
0507 08:12:14.698710 (+3667561us) ts_tablet_manager.cc:1120] Deleting tablet 924d8d2fe2f148cba13f107e4c70fc68
0507 08:12:14.698725 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.700905 (+ 2180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3666873}
I20240507 08:12:14.704846 5718 raft_consensus.cc:2454] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe673739d8084e7da5d7edd722e857d8 in term 1.
I20240507 08:12:14.704735 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1000) took 3673 ms. Trace:
I20240507 08:12:14.705821 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 4306751d16444ec5948c2ba5831063f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.706310 5840 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.706383 5517 leader_election.cc:302] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8; no voters:
I20240507 08:12:14.706557 5840 ts_tablet_manager.cc:1367] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.004s
I20240507 08:12:14.707194 5840 raft_consensus.cc:348] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.707429 5840 raft_consensus.cc:374] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.707512 5840 raft_consensus.cc:729] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.707768 5840 consensus_queue.cc:260] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.708146 6513 ts_tablet_manager.cc:1888] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.705767 5841 rpcz_store.cc:276] 0507 08:12:11.031380 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.031508 (+ 128us) service_pool.cc:225] Handling call
0507 08:12:14.703904 (+3672396us) ts_tablet_manager.cc:1120] Deleting tablet 128c2034d917473eb5c496319fdc9625
0507 08:12:14.703914 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.704716 (+ 802us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3671968}
I20240507 08:12:14.709998 7329 leader_election.cc:288] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.710160 7329 raft_consensus.cc:2781] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.710269 7329 raft_consensus.cc:686] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Becoming Leader. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Running, Role: LEADER
I20240507 08:12:14.710502 7329 consensus_queue.cc:237] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.711374 5520 leader_election.cc:341] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 79524df2415146ab926425c42068626c
I20240507 08:12:14.713766 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1001) took 3682 ms. Trace:
I20240507 08:12:14.714355 5841 rpcz_store.cc:276] 0507 08:12:11.031648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.031721 (+ 73us) service_pool.cc:225] Handling call
0507 08:12:14.710479 (+3678758us) ts_tablet_manager.cc:1120] Deleting tablet 5c49745fce2949d1909baf7583b157a7
0507 08:12:14.710487 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.713748 (+ 3261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3678555}
W20240507 08:12:14.715415 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42a34eaa8a514f349f1a092abfff155f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.716341 5841 tablet_replica.cc:330] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:14.717404 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e7a8a764bfdf4f358f30d679d0bbf5d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.718008 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7a8a764bfdf4f358f30d679d0bbf5d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.691000 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 565ac6091f5e48b1b024094d5560e275 (table default.Table_457ae0dd8b974005872846c9fe575b10 [id=796006aacf9741f8a4dc1d5916887655]) successfully deleted
W20240507 08:12:14.719802 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9ccbe2e368d14b56859d62d1cc742f90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9ccbe2e368d14b56859d62d1cc742f90 already in progress: creating tablet
W20240507 08:12:14.720007 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 924d8d2fe2f148cba13f107e4c70fc68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 924d8d2fe2f148cba13f107e4c70fc68 already in progress: creating tablet
I20240507 08:12:14.720047 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 4306751d16444ec5948c2ba5831063f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:14.720180 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 128c2034d917473eb5c496319fdc9625 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 128c2034d917473eb5c496319fdc9625 already in progress: creating tablet
W20240507 08:12:14.720357 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5c49745fce2949d1909baf7583b157a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c49745fce2949d1909baf7583b157a7 already in progress: creating tablet
I20240507 08:12:14.720342 5841 raft_consensus.cc:2229] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.721836 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 565ac6091f5e48b1b024094d5560e275 (table default.Table_457ae0dd8b974005872846c9fe575b10 [id=796006aacf9741f8a4dc1d5916887655]) successfully deleted
I20240507 08:12:14.721952 5841 raft_consensus.cc:2258] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.711057 5840 ts_tablet_manager.cc:1398] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:14.723183 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 4306751d16444ec5948c2ba5831063f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.725682 5840 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:14.726398 5656 consensus_peers.cc:485] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.727893 7318 tablet.cc:1722] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:14.731981 7333 consensus_queue.cc:1035] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.732569 7351 tablet.cc:1722] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:14.733878 5840 tablet_bootstrap.cc:654] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.734462 6513 log.cc:1196] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/35e34b60b78b402faf51e003c97db787
I20240507 08:12:14.737015 7408 raft_consensus.cc:482] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.737136 7408 raft_consensus.cc:504] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.737721 7408 leader_election.cc:288] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.738029 5841 ts_tablet_manager.cc:1875] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.738685 6513 ts_tablet_manager.cc:1909] T 35e34b60b78b402faf51e003c97db787 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.738715 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:14.739193 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4a5ad85280b49289eeef7eea3c2849d" } modify_external_catalogs: true
I20240507 08:12:14.739444 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f4a5ad85280b49289eeef7eea3c2849d" } modify_external_catalogs: true
W20240507 08:12:14.739607 5377 leader_election.cc:341] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.744606 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_087f9c936f35417f854ffeeb097ca835"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:14.745553 5839 tablet_bootstrap.cc:492] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.745772 5839 ts_tablet_manager.cc:1367] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.077s user 0.007s sys 0.004s
W20240507 08:12:14.745842 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_087f9c936f35417f854ffeeb097ca835 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.746608 5839 raft_consensus.cc:348] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.746868 5839 raft_consensus.cc:374] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.746976 5839 raft_consensus.cc:729] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.747263 5839 consensus_queue.cc:260] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.747668 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 992) took 4040 ms. Trace:
I20240507 08:12:14.748858 5843 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.749101 5843 ts_tablet_manager.cc:1367] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.007s
I20240507 08:12:14.748520 6513 rpcz_store.cc:276] 0507 08:12:10.707241 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707337 (+ 96us) service_pool.cc:225] Handling call
0507 08:12:14.655819 (+3948482us) ts_tablet_manager.cc:1120] Deleting tablet 35e34b60b78b402faf51e003c97db787
0507 08:12:14.655830 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.688168 (+ 32338us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.708038 (+ 19870us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.747325 (+ 39287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3948262}
I20240507 08:12:14.750902 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 35e34b60b78b402faf51e003c97db787 (table default.Table_7007a63c36ea47c78769b5bcafec4cf9 [id=71fd821e13734824a24f76fa2c220f1a]) successfully deleted
I20240507 08:12:14.751346 6513 tablet_replica.cc:330] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.752792 6513 raft_consensus.cc:2229] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:14.754791 5654 consensus_peers.cc:485] T 2754fc46c36346f9828198e3bb6ed3a7 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:14.754956 5654 consensus_peers.cc:485] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.755894 6513 raft_consensus.cc:2258] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.758141 5843 raft_consensus.cc:348] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.758399 5843 raft_consensus.cc:374] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.758498 5843 raft_consensus.cc:729] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.758800 5843 consensus_queue.cc:260] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.759369 5843 ts_tablet_manager.cc:1398] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240507 08:12:14.759953 5843 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.760023 6513 ts_tablet_manager.cc:1875] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.761418 7333 consensus_queue.cc:1035] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.764348 5839 ts_tablet_manager.cc:1398] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20240507 08:12:14.766398 5839 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.770974 5839 tablet_bootstrap.cc:654] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.772784 5840 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.772982 5840 ts_tablet_manager.cc:1367] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20240507 08:12:14.773998 5840 raft_consensus.cc:348] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.774346 5840 raft_consensus.cc:374] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.774443 5840 raft_consensus.cc:729] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.774762 5843 tablet_bootstrap.cc:654] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.774730 5840 consensus_queue.cc:260] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.775051 7333 raft_consensus.cc:482] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.775395 7333 raft_consensus.cc:504] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.776715 7333 leader_election.cc:288] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:14.777683 5656 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f2de26190bc44916809856d84ad68345
I20240507 08:12:14.775388 5840 ts_tablet_manager.cc:1398] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:14.778187 5840 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.772341 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1015) took 2057 ms. Trace:
I20240507 08:12:14.776674 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de26190bc44916809856d84ad68345" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:14.777001 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 70638.4us
I20240507 08:12:14.780138 7079 rpcz_store.cc:276] 0507 08:12:12.715293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.718526 (+ 3233us) service_pool.cc:225] Handling call
0507 08:12:14.689178 (+1970652us) ts_tablet_manager.cc:1120] Deleting tablet a3535f1fbfe34f6e95e9895771ae1350
0507 08:12:14.753712 (+ 64534us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.772323 (+ 18611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1870337}
W20240507 08:12:14.780539 5654 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.780491 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.780634 5654 leader_election.cc:302] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:14.780793 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a3535f1fbfe34f6e95e9895771ae1350 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3535f1fbfe34f6e95e9895771ae1350 already in progress: creating tablet
I20240507 08:12:14.780830 7394 raft_consensus.cc:2726] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.781108 7079 tablet_replica.cc:330] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:14.781066 5378 leader_election.cc:341] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.781211 5378 leader_election.cc:302] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.781353 5840 tablet_bootstrap.cc:654] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.781381 7079 raft_consensus.cc:2229] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.781522 7079 raft_consensus.cc:2258] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.781630 7408 raft_consensus.cc:2726] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.782091 7079 ts_tablet_manager.cc:1875] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.783195 7408 raft_consensus.cc:482] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.783305 7408 raft_consensus.cc:504] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.783852 7408 leader_election.cc:288] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.784373 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a30bfc61b4ff2be9f2418ca59b030" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:14.784619 5718 raft_consensus.cc:2454] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:14.784844 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a30bfc61b4ff2be9f2418ca59b030" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.785310 5377 leader_election.cc:302] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.785531 7408 raft_consensus.cc:2781] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.785624 7408 raft_consensus.cc:482] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.785681 7408 raft_consensus.cc:3037] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:14.788017 7408 raft_consensus.cc:504] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.788748 5378 leader_election.cc:341] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.788913 7408 leader_election.cc:288] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.789093 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a30bfc61b4ff2be9f2418ca59b030" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.789299 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "055a30bfc61b4ff2be9f2418ca59b030" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:14.789465 5718 raft_consensus.cc:3037] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:14.791947 5654 consensus_peers.cc:485] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.792426 5580 raft_consensus.cc:1261] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: Refusing update from remote peer 1cbb887175f345ada2433adaec906fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240507 08:12:14.791960 5718 raft_consensus.cc:2454] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:14.793553 5377 leader_election.cc:302] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:14.793558 7358 consensus_queue.cc:1035] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.793819 7408 raft_consensus.cc:2781] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.793936 7408 raft_consensus.cc:686] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:14.794175 7408 consensus_queue.cc:237] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.795842 5378 leader_election.cc:341] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.799435 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d75fc8323e274f5bb918bbcc918a336e" } modify_external_catalogs: true
I20240507 08:12:14.800034 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d75fc8323e274f5bb918bbcc918a336e" } modify_external_catalogs: true
I20240507 08:12:14.801999 7422 raft_consensus.cc:482] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.802431 7422 raft_consensus.cc:504] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:14.803364 7422 leader_election.cc:288] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:14.803664 5520 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:14.806782 5843 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:14.807439 5517 leader_election.cc:341] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:14.807610 5517 leader_election.cc:302] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:14.807873 7418 raft_consensus.cc:2726] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.807783 5843 ts_tablet_manager.cc:1367] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20240507 08:12:14.815960 5839 tablet_bootstrap.cc:492] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.816152 5839 ts_tablet_manager.cc:1367] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.001s
I20240507 08:12:14.816334 5843 raft_consensus.cc:348] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.816953 5839 raft_consensus.cc:348] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.817221 5839 raft_consensus.cc:374] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.817313 5839 raft_consensus.cc:729] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.817564 5839 consensus_queue.cc:260] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.818140 5839 ts_tablet_manager.cc:1398] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:14.818256 5843 raft_consensus.cc:374] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.818423 5839 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.818509 5843 raft_consensus.cc:729] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.821710 7079 ts_tablet_manager.cc:1888] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:14.821931 5843 consensus_queue.cc:260] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.822607 5843 ts_tablet_manager.cc:1398] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240507 08:12:14.822885 5843 tablet_bootstrap.cc:492] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:14.823001 7079 log.cc:1196] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/03a59c7257e74b4abaa4ea7a1ee9136d
I20240507 08:12:14.824535 5843 tablet_bootstrap.cc:654] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.824952 7079 ts_tablet_manager.cc:1909] T 03a59c7257e74b4abaa4ea7a1ee9136d P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:14.825341 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1012) took 2125 ms. Trace:
I20240507 08:12:14.825510 7079 rpcz_store.cc:276] 0507 08:12:12.699634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.721236 (+ 21602us) service_pool.cc:225] Handling call
0507 08:12:14.781053 (+2059817us) ts_tablet_manager.cc:1120] Deleting tablet 03a59c7257e74b4abaa4ea7a1ee9136d
0507 08:12:14.781067 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.789707 (+ 8640us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.821611 (+ 31904us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.825324 (+ 3713us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1961733}
I20240507 08:12:14.823197 5839 tablet_bootstrap.cc:654] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.826053 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 03a59c7257e74b4abaa4ea7a1ee9136d (table default.Table_e5179c5e5c224b3690b690b725a01cd0 [id=ab3d4a967b1349a988bcd67feb187096]) successfully deleted
I20240507 08:12:14.827291 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1016) took 2088 ms. Trace:
I20240507 08:12:14.828008 7079 rpcz_store.cc:276] 0507 08:12:12.738929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.739007 (+ 78us) service_pool.cc:225] Handling call
0507 08:12:14.827162 (+2088155us) ts_tablet_manager.cc:1120] Deleting tablet c1932ffe7d37455c8aa6bb83ffd11498
0507 08:12:14.827171 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.827273 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008003}
I20240507 08:12:14.828393 7079 tablet_replica.cc:330] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:14.828573 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c1932ffe7d37455c8aa6bb83ffd11498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1932ffe7d37455c8aa6bb83ffd11498 already in progress: creating tablet
I20240507 08:12:14.830502 7333 consensus_queue.cc:1035] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:14.807544 5841 ts_tablet_manager.cc:1888] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.832430 5841 log.cc:1196] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/31251576beee4a8a81a453b721c9c058
I20240507 08:12:14.847589 5840 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.847901 5841 ts_tablet_manager.cc:1909] T 31251576beee4a8a81a453b721c9c058 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:14.851054 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 093fb3986a674b93a435408c7ec85500
I20240507 08:12:14.852353 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 093fb3986a674b93a435408c7ec85500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.851783 5840 ts_tablet_manager.cc:1367] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.001s
I20240507 08:12:14.853888 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1002) took 3822 ms. Trace:
I20240507 08:12:14.853567 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 093fb3986a674b93a435408c7ec85500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.855077 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f2867d292c9d45a3b77c426279d29c32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.854458 5656 consensus_peers.cc:485] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6e68364238c4b66b8c0de0f4a2ea855. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:14.855666 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2867d292c9d45a3b77c426279d29c32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:14.857234 5654 consensus_peers.cc:485] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.855683 5841 rpcz_store.cc:276] 0507 08:12:11.031911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.031965 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:14.716063 (+3684098us) ts_tablet_manager.cc:1120] Deleting tablet 31251576beee4a8a81a453b721c9c058
0507 08:12:14.716073 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.771511 (+ 55438us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.807440 (+ 35929us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.853867 (+ 46427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3683861}
I20240507 08:12:14.858856 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 998) took 3828 ms. Trace:
I20240507 08:12:14.862543 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_47367ed796754576afd6b024153d847f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.863601 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47367ed796754576afd6b024153d847f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.870422 6513 ts_tablet_manager.cc:1888] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:14.860551 5841 rpcz_store.cc:276] 0507 08:12:11.030804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.032243 (+ 1439us) service_pool.cc:225] Handling call
0507 08:12:14.858732 (+3826489us) ts_tablet_manager.cc:1120] Deleting tablet 39e5e67df91f4029bf55e5a82dc31f82
0507 08:12:14.858742 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.858838 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3826309}
I20240507 08:12:14.873257 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 997) took 3842 ms. Trace:
I20240507 08:12:14.872776 5840 raft_consensus.cc:348] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.876212 6513 log.cc:1196] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/0d617213b959452ab6519809e56db128
I20240507 08:12:14.875774 5840 raft_consensus.cc:374] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.876900 5840 raft_consensus.cc:729] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.875833 5841 rpcz_store.cc:276] 0507 08:12:11.030425 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.035376 (+ 4951us) service_pool.cc:225] Handling call
0507 08:12:14.872500 (+3837124us) ts_tablet_manager.cc:1120] Deleting tablet 81fcf82e60c94e9eaadd75bd37595261
0507 08:12:14.873078 (+ 578us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.873235 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3836909}
I20240507 08:12:14.881973 6513 ts_tablet_manager.cc:1909] T 0d617213b959452ab6519809e56db128 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.882455 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1005) took 3776 ms. Trace:
I20240507 08:12:14.882606 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0a81eec1a1d34fa8bdbe9fac1d360516"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.883306 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a81eec1a1d34fa8bdbe9fac1d360516 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.891265 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_adbdf44307174a5b8a7a916c5fc6c5bb" } new_table_name: "default.Table_1f79d70804784363a637492331c96f9c" modify_external_catalogs: true
I20240507 08:12:14.888356 7378 raft_consensus.cc:482] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.891631 7378 raft_consensus.cc:504] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.892531 7378 leader_election.cc:288] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:14.893757 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:14.894526 5378 leader_election.cc:341] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.885142 5840 consensus_queue.cc:260] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.893363 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:14.895772 5377 leader_election.cc:341] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.895902 5377 leader_election.cc:302] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.896508 7378 raft_consensus.cc:2726] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:14.897383 7079 raft_consensus.cc:2229] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:14.897825 7079 pending_rounds.cc:62] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Trying to abort 2 pending ops.
I20240507 08:12:14.897925 7079 pending_rounds.cc:68] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7024924810968436736 op_type: NO_OP noop_request { }
I20240507 08:12:14.898078 7079 raft_consensus.cc:2866] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:14.898183 7079 pending_rounds.cc:68] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924793962274816 op_type: NO_OP noop_request { }
I20240507 08:12:14.898324 7079 raft_consensus.cc:2866] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:14.896744 5841 rpcz_store.cc:276] 0507 08:12:11.105758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.109635 (+ 3877us) service_pool.cc:225] Handling call
0507 08:12:14.882294 (+3772659us) ts_tablet_manager.cc:1120] Deleting tablet 7fbe0f58bad541e7bcfb611105328d56
0507 08:12:14.882309 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.882436 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3772462}
I20240507 08:12:14.898444 7079 raft_consensus.cc:2258] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.901233 5840 ts_tablet_manager.cc:1398] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.046s user 0.002s sys 0.001s
I20240507 08:12:14.901697 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1004) took 3796 ms. Trace:
I20240507 08:12:14.900801 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 994) took 4193 ms. Trace:
I20240507 08:12:14.902191 6513 rpcz_store.cc:276] 0507 08:12:10.707422 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707457 (+ 35us) service_pool.cc:225] Handling call
0507 08:12:14.750775 (+4043318us) ts_tablet_manager.cc:1120] Deleting tablet 0d617213b959452ab6519809e56db128
0507 08:12:14.750786 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.769489 (+ 18703us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.870284 (+100795us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.900579 (+ 30295us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4043174}
I20240507 08:12:14.903800 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 19b77b6ef778454e91105e1e9b60020d
I20240507 08:12:14.904311 5258 catalog_manager.cc:6554] ReplaceTablet: tablet 19b77b6ef778454e91105e1e9b60020d deleted and replaced by tablet dd97542e3db0413c8272ff2971801be7
I20240507 08:12:14.902001 5841 rpcz_store.cc:276] 0507 08:12:11.105611 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.110825 (+ 5214us) service_pool.cc:225] Handling call
0507 08:12:14.900294 (+3789469us) ts_tablet_manager.cc:1120] Deleting tablet 0c51ec8f693e4f108100c75470ead2a9
0507 08:12:14.900308 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.901549 (+ 1241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3785250}
I20240507 08:12:14.904719 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_44eb65d8c630424989eb6b54fbd21962" } new_table_name: "default.Table_999adbd340214baeae51dc58e413ef3b" modify_external_catalogs: true
I20240507 08:12:14.905475 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1014) took 3533 ms. Trace:
I20240507 08:12:14.906359 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 0d617213b959452ab6519809e56db128 (table default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6 [id=bce0bd66e7b74db7934937410c81b20b]) successfully deleted
I20240507 08:12:14.906528 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b77b6ef778454e91105e1e9b60020d with delete_type TABLET_DATA_DELETED (replaced by tablet dd97542e3db0413c8272ff2971801be7) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:14.905088 5840 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:14.906528 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 993) took 4197 ms. Trace:
I20240507 08:12:14.906834 6513 rpcz_store.cc:276] 0507 08:12:10.707315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.707612 (+ 297us) service_pool.cc:225] Handling call
0507 08:12:14.904735 (+4197123us) ts_tablet_manager.cc:1120] Deleting tablet 9ced879f35e14106b2609c78b7f24002
0507 08:12:14.904749 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.904837 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196975}
I20240507 08:12:14.907158 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 995) took 4196 ms. Trace:
I20240507 08:12:14.907248 6513 rpcz_store.cc:276] 0507 08:12:10.710479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.710524 (+ 45us) service_pool.cc:225] Handling call
0507 08:12:14.907077 (+4196553us) ts_tablet_manager.cc:1120] Deleting tablet 9e447562e82347a584e39586500c27c2
0507 08:12:14.907087 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.907143 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196407}
I20240507 08:12:14.907414 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b77b6ef778454e91105e1e9b60020d with delete_type TABLET_DATA_DELETED (replaced by tablet dd97542e3db0413c8272ff2971801be7) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:14.907505 6513 tablet_replica.cc:330] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:14.907696 6513 raft_consensus.cc:2229] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 2 LEADER]: Raft consensus shutting down.
W20240507 08:12:14.907714 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9ced879f35e14106b2609c78b7f24002 because the tablet was not found. No further retry: Not found: Tablet not found: 9ced879f35e14106b2609c78b7f24002
W20240507 08:12:14.907979 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9e447562e82347a584e39586500c27c2 because the tablet was not found. No further retry: Not found: Tablet not found: 9e447562e82347a584e39586500c27c2
I20240507 08:12:14.908033 6513 raft_consensus.cc:2258] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:14.907934 5841 rpcz_store.cc:276] 0507 08:12:11.372440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.372484 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:14.905336 (+3532852us) ts_tablet_manager.cc:1120] Deleting tablet e561c811a1de48f9bc91175a3d22805a
0507 08:12:14.905350 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.905456 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3532657}
I20240507 08:12:14.910332 6513 ts_tablet_manager.cc:1875] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.911216 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1012) took 3539 ms. Trace:
W20240507 08:12:14.913947 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9ff253ff827f4694bd27994825793108 consensus state has no leader
I20240507 08:12:14.914184 5841 rpcz_store.cc:276] 0507 08:12:11.372216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.372863 (+ 647us) service_pool.cc:225] Handling call
0507 08:12:14.910331 (+3537468us) ts_tablet_manager.cc:1120] Deleting tablet a6c50d46db8949ee978b87f83a640c0a
0507 08:12:14.910623 (+ 292us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.911198 (+ 575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3537215}
I20240507 08:12:14.916222 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1013) took 3543 ms. Trace:
I20240507 08:12:14.917034 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7a8bd4e6c9d149189ae4d0f6877c8cdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.917640 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a8bd4e6c9d149189ae4d0f6877c8cdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.916788 5841 rpcz_store.cc:276] 0507 08:12:11.372335 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.377876 (+ 5541us) service_pool.cc:225] Handling call
0507 08:12:14.915667 (+3537791us) ts_tablet_manager.cc:1120] Deleting tablet a903c17e01c241bcb13787ef551395d2
0507 08:12:14.915679 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.916202 (+ 523us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3537296}
I20240507 08:12:14.920774 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1018) took 3438 ms. Trace:
I20240507 08:12:14.925849 5841 rpcz_store.cc:276] 0507 08:12:11.482483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.484149 (+ 1666us) service_pool.cc:225] Handling call
0507 08:12:14.919715 (+3435566us) ts_tablet_manager.cc:1120] Deleting tablet b6b9232426b948ae8634f7afd65df393
0507 08:12:14.919729 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.920758 (+ 1029us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3435112}
I20240507 08:12:14.927124 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2f032a109d8d4902bd1611542a32e9d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.927659 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f032a109d8d4902bd1611542a32e9d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.920552 5840 tablet_bootstrap.cc:654] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.923055 6513 ts_tablet_manager.cc:1888] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240507 08:12:14.928490 6513 log.cc:1196] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:14.928898 6513 ts_tablet_manager.cc:1909] T 1037c3c0af5e464bb23a311e552fcf21 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:14.934082 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1019) took 3441 ms. Trace:
I20240507 08:12:14.936141 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6571d5b9dda642549111e4a278c03f33" } new_table_name: "default.Table_e6f60f9cdfa148418c1d828dcbe9b866" modify_external_catalogs: true
I20240507 08:12:14.935858 5841 rpcz_store.cc:276] 0507 08:12:11.492411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.493527 (+ 1116us) service_pool.cc:225] Handling call
0507 08:12:14.928630 (+3435103us) ts_tablet_manager.cc:1120] Deleting tablet 482e4d0e762b488b8a0cc4e81c2c821d
0507 08:12:14.928643 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.934053 (+ 5410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3434505}
I20240507 08:12:14.936659 7100 raft_consensus.cc:482] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:14.936789 7100 raft_consensus.cc:504] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.937435 7100 leader_election.cc:288] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.937922 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:14.938042 5841 tablet_replica.cc:330] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:14.938496 5654 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.938356 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:14.939044 5446 raft_consensus.cc:2454] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:14.939532 5656 leader_election.cc:302] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.940107 7100 raft_consensus.cc:2781] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:14.940196 7100 raft_consensus.cc:482] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:14.940264 7100 raft_consensus.cc:3037] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:14.942525 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f975fac7312c4de39fbeada171119ed7 consensus state has no leader
I20240507 08:12:14.942750 7100 raft_consensus.cc:504] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.943313 7100 leader_election.cc:288] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:14.943259 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_97f7aa309ac14b62b69cbe42ce8ff48b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:14.943749 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97f7aa309ac14b62b69cbe42ce8ff48b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:14.943740 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:14.944844 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84212d6cb2c742c1a0f3fdae8be8ebe1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
W20240507 08:12:14.944945 5656 consensus_peers.cc:485] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2df872d04b94c30872b03367df3cc31. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.945063 5437 raft_consensus.cc:3037] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:14.946851 5654 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:14.949615 5437 raft_consensus.cc:2454] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:14.950398 5654 consensus_peers.cc:485] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.951337 5656 leader_election.cc:302] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:14.951903 7330 raft_consensus.cc:2781] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:14.952023 7330 raft_consensus.cc:686] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:14.952256 7330 consensus_queue.cc:237] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.952555 5840 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:14.953552 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 093fb3986a674b93a435408c7ec85500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.953910 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 31251576beee4a8a81a453b721c9c058 (table default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]) successfully deleted
W20240507 08:12:14.954191 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 39e5e67df91f4029bf55e5a82dc31f82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 39e5e67df91f4029bf55e5a82dc31f82 already in progress: creating tablet
W20240507 08:12:14.954466 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 81fcf82e60c94e9eaadd75bd37595261 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81fcf82e60c94e9eaadd75bd37595261 already in progress: creating tablet
W20240507 08:12:14.954692 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7fbe0f58bad541e7bcfb611105328d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbe0f58bad541e7bcfb611105328d56 already in progress: creating tablet
W20240507 08:12:14.954946 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0c51ec8f693e4f108100c75470ead2a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c51ec8f693e4f108100c75470ead2a9 already in progress: creating tablet
I20240507 08:12:14.955036 5841 raft_consensus.cc:2229] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:14.955153 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e561c811a1de48f9bc91175a3d22805a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e561c811a1de48f9bc91175a3d22805a already in progress: creating tablet
I20240507 08:12:14.955391 5841 raft_consensus.cc:2258] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:14.955403 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a6c50d46db8949ee978b87f83a640c0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c50d46db8949ee978b87f83a640c0a already in progress: creating tablet
W20240507 08:12:14.955601 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a903c17e01c241bcb13787ef551395d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a903c17e01c241bcb13787ef551395d2 already in progress: creating tablet
W20240507 08:12:14.955837 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b6b9232426b948ae8634f7afd65df393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6b9232426b948ae8634f7afd65df393 already in progress: creating tablet
W20240507 08:12:14.956111 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 482e4d0e762b488b8a0cc4e81c2c821d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 482e4d0e762b488b8a0cc4e81c2c821d already in progress: creating tablet
I20240507 08:12:14.957659 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b77b6ef778454e91105e1e9b60020d with delete_type TABLET_DATA_DELETED (replaced by tablet dd97542e3db0413c8272ff2971801be7) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:14.959000 5839 tablet_bootstrap.cc:492] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:14.959201 5839 ts_tablet_manager.cc:1367] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.001s
I20240507 08:12:14.960054 5839 raft_consensus.cc:348] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.960323 5839 raft_consensus.cc:374] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.960466 5839 raft_consensus.cc:729] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.960814 5839 consensus_queue.cc:260] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:14.961668 5839 ts_tablet_manager.cc:1398] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:14.968384 5840 ts_tablet_manager.cc:1367] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.003s
I20240507 08:12:14.972422 5261 catalog_manager.cc:5484] T ba28c36f91e141f4a1372e7544b44281 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:14.969836 5841 ts_tablet_manager.cc:1875] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.973541 5261 catalog_manager.cc:5484] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:14.974421 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 3 got 0
I20240507 08:12:14.974997 5261 catalog_manager.cc:5484] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:14.976300 7079 ts_tablet_manager.cc:1875] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:14.980881 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]). Expected version 1 got 0
I20240507 08:12:14.986711 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:14.990475 5839 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:14.992019 5839 tablet_bootstrap.cc:654] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:14.992283 5840 raft_consensus.cc:348] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.994586 5656 consensus_peers.cc:485] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e298f11ff9f54e48b8de435f6849f39a. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.995481 5843 tablet_bootstrap.cc:492] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:14.992671 5840 raft_consensus.cc:374] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:14.995604 5840 raft_consensus.cc:729] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:14.996029 5840 consensus_queue.cc:260] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:14.997712 5654 consensus_peers.cc:485] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:14.998246 5843 ts_tablet_manager.cc:1367] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.175s user 0.001s sys 0.002s
I20240507 08:12:14.999415 5840 ts_tablet_manager.cc:1398] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
I20240507 08:12:15.001080 5843 raft_consensus.cc:348] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.001364 5843 raft_consensus.cc:374] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.001515 5843 raft_consensus.cc:729] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.001847 5843 consensus_queue.cc:260] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.002036 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 996) took 4291 ms. Trace:
I20240507 08:12:15.002123 6513 rpcz_store.cc:276] 0507 08:12:10.710858 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.710898 (+ 40us) service_pool.cc:225] Handling call
0507 08:12:14.907461 (+4196563us) ts_tablet_manager.cc:1120] Deleting tablet 1037c3c0af5e464bb23a311e552fcf21
0507 08:12:14.907471 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.915043 (+ 7572us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:14.922960 (+ 7917us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.002017 (+ 79057us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196417}
I20240507 08:12:15.002419 6513 tablet_replica.cc:330] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:15.002584 6513 raft_consensus.cc:2229] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:15.002902 6513 raft_consensus.cc:2258] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.003402 6513 ts_tablet_manager.cc:1875] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.003401 7418 raft_consensus.cc:482] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.003525 7418 raft_consensus.cc:504] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.004148 7418 leader_election.cc:288] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:15.005141 5517 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 0d617213b959452ab6519809e56db128
W20240507 08:12:15.007824 5520 leader_election.cc:341] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 0d617213b959452ab6519809e56db128
I20240507 08:12:15.007941 5520 leader_election.cc:302] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:15.008320 7418 raft_consensus.cc:2726] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:15.008847 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1037c3c0af5e464bb23a311e552fcf21 (table default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]) successfully deleted
I20240507 08:12:15.009796 6513 ts_tablet_manager.cc:1888] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.010004 6513 log.cc:1196] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1c7c39ec9fec4331afabb39d1483ef57
I20240507 08:12:15.010071 5839 tablet_bootstrap.cc:492] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:15.010259 5839 ts_tablet_manager.cc:1367] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.001s
I20240507 08:12:15.010445 5258 master_service.cc:946] ReplaceTablet: received request to replace tablet f748727d3a88442d90cf81bae48acd36 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:15.010583 6513 ts_tablet_manager.cc:1909] T 1c7c39ec9fec4331afabb39d1483ef57 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:15.010905 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:15.011145 5843 ts_tablet_manager.cc:1398] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20240507 08:12:15.011693 5839 raft_consensus.cc:348] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.011922 5839 raft_consensus.cc:374] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.012008 5839 raft_consensus.cc:729] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.012157 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 997) took 4300 ms. Trace:
I20240507 08:12:15.012259 5839 consensus_queue.cc:260] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.012355 6513 rpcz_store.cc:276] 0507 08:12:10.711229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.711272 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:15.002375 (+4291103us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:12:15.002384 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.006840 (+ 4456us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.009702 (+ 2862us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.012138 (+ 2436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4290966}
I20240507 08:12:15.013113 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 998) took 4301 ms. Trace:
I20240507 08:12:15.013300 6513 rpcz_store.cc:276] 0507 08:12:10.711573 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.711611 (+ 38us) service_pool.cc:225] Handling call
0507 08:12:15.012977 (+4301366us) ts_tablet_manager.cc:1120] Deleting tablet 077ccc3946b34008bea7b80d8aee2662
0507 08:12:15.012986 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.013098 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4301223}
I20240507 08:12:15.013797 7330 raft_consensus.cc:482] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.013965 7330 raft_consensus.cc:504] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.014918 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:15.015369 5654 leader_election.cc:341] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.015489 7079 ts_tablet_manager.cc:1888] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20240507 08:12:15.015604 7079 log.cc:1196] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:15.015944 7079 ts_tablet_manager.cc:1909] T 1037c3c0af5e464bb23a311e552fcf21 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:15.016387 7330 leader_election.cc:288] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.016633 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.016866 5447 raft_consensus.cc:2454] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.017283 5656 leader_election.cc:302] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.017788 7330 raft_consensus.cc:2781] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.017903 7330 raft_consensus.cc:482] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.018014 7330 raft_consensus.cc:3037] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.020499 7330 raft_consensus.cc:504] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.021909 5843 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.021996 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.023643 5840 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:15.024947 5654 leader_election.cc:341] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.026873 5841 ts_tablet_manager.cc:1888] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.027254 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1c7c39ec9fec4331afabb39d1483ef57 (table default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]) successfully deleted
W20240507 08:12:15.027554 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 077ccc3946b34008bea7b80d8aee2662 because the tablet was not found. No further retry: Not found: Tablet not found: 077ccc3946b34008bea7b80d8aee2662
I20240507 08:12:15.027778 5261 catalog_manager.cc:5484] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:15.028357 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_a63ec02af6ab4e85830582d44267f8c1 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 1 got 0
I20240507 08:12:15.029381 5839 ts_tablet_manager.cc:1398] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.019s user 0.000s sys 0.001s
W20240507 08:12:15.030292 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dcd7ec455826497893af43763c7e7acf consensus state has no leader
I20240507 08:12:15.031282 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 62d6eac6baa249eb847c5625d64f998f
I20240507 08:12:15.031749 5257 catalog_manager.cc:6554] ReplaceTablet: tablet 62d6eac6baa249eb847c5625d64f998f deleted and replaced by tablet 140b4601f1a043329c4de612a3ae244c
I20240507 08:12:15.031831 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 695b2e6529914f51ba1884a216adbeb5 (DEFAULT_TABLE table=default.Table_f46b82cdb45a45d9886098ab730da4df [id=10d35b20d52c4c8aa680c543d9ec1013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.032181 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695b2e6529914f51ba1884a216adbeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.032560 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]). Expected version 2 got 0
I20240507 08:12:15.036319 7330 leader_election.cc:288] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.037098 5839 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:15.038110 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6eac6baa249eb847c5625d64f998f with delete_type TABLET_DATA_DELETED (replaced by tablet 140b4601f1a043329c4de612a3ae244c) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.038411 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6eac6baa249eb847c5625d64f998f with delete_type TABLET_DATA_DELETED (replaced by tablet 140b4601f1a043329c4de612a3ae244c) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.037940 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 999) took 4326 ms. Trace:
I20240507 08:12:15.038584 6513 rpcz_store.cc:276] 0507 08:12:10.711921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.711959 (+ 38us) service_pool.cc:225] Handling call
0507 08:12:15.037508 (+4325549us) ts_tablet_manager.cc:1120] Deleting tablet 193ec0b463d04835b74df80229c85cde
0507 08:12:15.037522 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.037764 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4325168}
I20240507 08:12:15.042323 5839 tablet_bootstrap.cc:654] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.039891 5840 tablet_bootstrap.cc:654] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.043437 6513 tablet_replica.cc:330] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:15.044304 6513 raft_consensus.cc:2229] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.045872 6513 raft_consensus.cc:2258] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:15.048339 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c34a8ee0ca24e9490f3ba409b62a009 consensus state has no leader
I20240507 08:12:15.048743 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9e717d2407ba4a54af3626573fb38c0b" } modify_external_catalogs: true
I20240507 08:12:15.027495 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8ebbcf0d5a54d63a96a9374e839af83" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.030349 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 695b2e6529914f51ba1884a216adbeb5 (DEFAULT_TABLE table=default.Table_f46b82cdb45a45d9886098ab730da4df [id=10d35b20d52c4c8aa680c543d9ec1013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.052918 5447 raft_consensus.cc:3037] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.053207 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695b2e6529914f51ba1884a216adbeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.056008 5447 raft_consensus.cc:2454] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:15.034961 5843 tablet_bootstrap.cc:654] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.037468 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet 62d6eac6baa249eb847c5625d64f998f with delete_type TABLET_DATA_DELETED (replaced by tablet 140b4601f1a043329c4de612a3ae244c) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.056321 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9e717d2407ba4a54af3626573fb38c0b" } modify_external_catalogs: true
I20240507 08:12:15.056450 5656 leader_election.cc:302] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.056874 7330 raft_consensus.cc:2781] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:15.056941 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 193ec0b463d04835b74df80229c85cde because the tablet was not found. No further retry: Not found: Tablet not found: 193ec0b463d04835b74df80229c85cde
I20240507 08:12:15.057114 7330 raft_consensus.cc:686] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.057333 5841 log.cc:1196] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/957775c01be94cc081db344c95a81310
I20240507 08:12:15.057353 7330 consensus_queue.cc:237] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.057719 5841 ts_tablet_manager.cc:1909] T 957775c01be94cc081db344c95a81310 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.057746 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a24813c72abd41c3830c7392c6149151
I20240507 08:12:15.058589 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 (DEFAULT_TABLE table=default.Table_ee23488ce5a940559de6005aa58093f5 [id=96e1fa2df17f40978ca340b78d5df505]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.058701 5422 tablet_service.cc:1467] Processing CreateTablet for tablet f3b065c19b8d486d99689f9d7c7b9c25 (DEFAULT_TABLE table=default.Table_c0de414a14264debbc99648f2cdabb8f [id=6085aa7f9d7949ce82c8a3edfddbbefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.058940 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78c08d758043c78d5bb8281b7f2d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.059083 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 17276d2d36874841b0d41f0d1649a6b6 (DEFAULT_TABLE table=default.Table_16b922361cc4472890fd7edbe813b667 [id=eab00a727cb54d318b434c67c1405a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.059260 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 (DEFAULT_TABLE table=default.Table_ee23488ce5a940559de6005aa58093f5 [id=96e1fa2df17f40978ca340b78d5df505]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.059392 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276d2d36874841b0d41f0d1649a6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.059396 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 695b2e6529914f51ba1884a216adbeb5 (DEFAULT_TABLE table=default.Table_f46b82cdb45a45d9886098ab730da4df [id=10d35b20d52c4c8aa680c543d9ec1013]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.059602 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 17276d2d36874841b0d41f0d1649a6b6 (DEFAULT_TABLE table=default.Table_16b922361cc4472890fd7edbe813b667 [id=eab00a727cb54d318b434c67c1405a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.060847 5557 tablet_service.cc:1467] Processing CreateTablet for tablet f3b065c19b8d486d99689f9d7c7b9c25 (DEFAULT_TABLE table=default.Table_c0de414a14264debbc99648f2cdabb8f [id=6085aa7f9d7949ce82c8a3edfddbbefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.061293 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b065c19b8d486d99689f9d7c7b9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.061818 5698 tablet_service.cc:1467] Processing CreateTablet for tablet f3b065c19b8d486d99689f9d7c7b9c25 (DEFAULT_TABLE table=default.Table_c0de414a14264debbc99648f2cdabb8f [id=6085aa7f9d7949ce82c8a3edfddbbefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.061837 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 17276d2d36874841b0d41f0d1649a6b6 (DEFAULT_TABLE table=default.Table_16b922361cc4472890fd7edbe813b667 [id=eab00a727cb54d318b434c67c1405a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.061959 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2eeab823921445deb9bd071d18014442" } new_table_name: "default.Table_4671310a79474597a1090defb2046acd" modify_external_catalogs: true
I20240507 08:12:15.062183 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276d2d36874841b0d41f0d1649a6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.063119 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet db92da02e5af49f5a13a814f3ce4f9c3
I20240507 08:12:15.064090 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b065c19b8d486d99689f9d7c7b9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.058573 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1021) took 3408 ms. Trace:
I20240507 08:12:15.064898 5841 rpcz_store.cc:276] 0507 08:12:11.650063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.650121 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:14.937293 (+3287172us) ts_tablet_manager.cc:1120] Deleting tablet 957775c01be94cc081db344c95a81310
0507 08:12:14.937906 (+ 613us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.023890 (+ 85984us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.026793 (+ 2903us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.058555 (+ 31762us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15490,"tablet-delete.queue_time_us":3286878}
I20240507 08:12:15.065300 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1023) took 3415 ms. Trace:
I20240507 08:12:15.065371 5841 rpcz_store.cc:276] 0507 08:12:11.650270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.650323 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:15.065146 (+3414823us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:12:15.065154 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.065286 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3414666}
I20240507 08:12:15.065614 5841 tablet_replica.cc:330] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.065775 5841 raft_consensus.cc:2229] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.065750 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet a24813c72abd41c3830c7392c6149151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.065909 5841 raft_consensus.cc:2258] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.066403 5841 ts_tablet_manager.cc:1875] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.067248 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.067409 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 957775c01be94cc081db344c95a81310 (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
W20240507 08:12:15.067600 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1c7c39ec9fec4331afabb39d1483ef57 because the tablet was not found. No further retry: Not found: Tablet not found: 1c7c39ec9fec4331afabb39d1483ef57
I20240507 08:12:15.074767 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.076318 7408 raft_consensus.cc:482] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.076436 7408 raft_consensus.cc:504] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.077056 7408 leader_election.cc:288] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.077519 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a015bf84a454d49b6497c16ed15c84a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:15.078110 5378 leader_election.cc:341] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.080389 7418 raft_consensus.cc:482] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.080500 7418 raft_consensus.cc:504] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.081041 7418 leader_election.cc:288] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:15.081820 5654 consensus_peers.cc:485] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:15.081825 5520 leader_election.cc:341] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: d451d03349794726a71c255a7f177436
W20240507 08:12:15.082162 5517 leader_election.cc:341] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: d451d03349794726a71c255a7f177436
I20240507 08:12:15.082250 5517 leader_election.cc:302] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:15.082250 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a015bf84a454d49b6497c16ed15c84a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.082420 5718 raft_consensus.cc:2454] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:15.082635 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet a24813c72abd41c3830c7392c6149151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.082803 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet db92da02e5af49f5a13a814f3ce4f9c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.083261 7418 raft_consensus.cc:2726] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:15.086472 5377 leader_election.cc:302] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.086536 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 14b073d71e934788a3b3575b52692d7b (DEFAULT_TABLE table=default.Table_381633ec2ac14aa4a5325fd26dafde7d [id=1c935e09f2f24adda618fc8b84093061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.086902 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b073d71e934788a3b3575b52692d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.090891 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 14b073d71e934788a3b3575b52692d7b (DEFAULT_TABLE table=default.Table_381633ec2ac14aa4a5325fd26dafde7d [id=1c935e09f2f24adda618fc8b84093061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.090958 5425 tablet_service.cc:1467] Processing CreateTablet for tablet fdffec6ce0da4c409466783bd76a6b4b (DEFAULT_TABLE table=default.Table_bbe582048e334931902cc697661690ca [id=d43b1125e24e4e79baffc5a279a81308]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.091145 5427 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3 (DEFAULT_TABLE table=default.Table_a668c8aa1ac34ee799dac4150b2e59ec [id=d1cc3ddd685344628315c7e0024b82ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.091199 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 1652772fff2d4ed9b50bb3022623eaa8 (DEFAULT_TABLE table=default.Table_5b0f0f6f249a4ceb83421dc56c43921e [id=34493ee6856149f283688904241d7df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.091293 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 1652772fff2d4ed9b50bb3022623eaa8 (DEFAULT_TABLE table=default.Table_5b0f0f6f249a4ceb83421dc56c43921e [id=34493ee6856149f283688904241d7df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.091353 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdffec6ce0da4c409466783bd76a6b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.090958 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79d55827864491a6eddac99e383ba8 (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.091794 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.092015 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1652772fff2d4ed9b50bb3022623eaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.092262 5689 tablet_service.cc:1467] Processing CreateTablet for tablet fdffec6ce0da4c409466783bd76a6b4b (DEFAULT_TABLE table=default.Table_bbe582048e334931902cc697661690ca [id=d43b1125e24e4e79baffc5a279a81308]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.093822 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 24e3ae77a7704423946b9074c19cb30d (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.094125 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e3ae77a7704423946b9074c19cb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.097219 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79d55827864491a6eddac99e383ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.099016 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79d55827864491a6eddac99e383ba8 (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.099367 5691 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3 (DEFAULT_TABLE table=default.Table_a668c8aa1ac34ee799dac4150b2e59ec [id=d1cc3ddd685344628315c7e0024b82ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.099707 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 24e3ae77a7704423946b9074c19cb30d (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.100498 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7c08a4452a324ca5b3cc664cfef163ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.101089 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c08a4452a324ca5b3cc664cfef163ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.104598 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet a24813c72abd41c3830c7392c6149151 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.105062 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 (DEFAULT_TABLE table=default.Table_ee23488ce5a940559de6005aa58093f5 [id=96e1fa2df17f40978ca340b78d5df505]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.105491 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78c08d758043c78d5bb8281b7f2d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.105710 7330 raft_consensus.cc:482] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.105861 7330 raft_consensus.cc:504] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.106494 7330 leader_election.cc:288] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.107919 7330 raft_consensus.cc:482] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.108078 7330 raft_consensus.cc:504] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.109133 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.109344 5437 raft_consensus.cc:2454] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.109776 5656 leader_election.cc:302] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.110108 7394 raft_consensus.cc:2781] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.110178 7394 raft_consensus.cc:482] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.110230 7394 raft_consensus.cc:3037] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.112027 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.112185 5447 raft_consensus.cc:2454] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.112588 7394 raft_consensus.cc:504] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.112715 5656 leader_election.cc:302] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.112936 7358 raft_consensus.cc:2781] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.112994 7358 raft_consensus.cc:482] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.113050 7358 raft_consensus.cc:3037] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.113165 7394 leader_election.cc:288] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.113970 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.114115 5447 raft_consensus.cc:3037] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.115432 7358 raft_consensus.cc:504] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.115964 7358 leader_election.cc:288] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.116549 5447 raft_consensus.cc:2454] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:15.117062 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.117398 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.117563 7330 leader_election.cc:288] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.117713 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.118201 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.118736 5656 leader_election.cc:302] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.118999 7330 raft_consensus.cc:2781] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.119158 7330 raft_consensus.cc:686] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.119515 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b9429e5cc274198974b27c40c7e4e37" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.119675 5447 raft_consensus.cc:3037] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.119772 5585 raft_consensus.cc:3037] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.122551 5447 raft_consensus.cc:2454] T 3b9429e5cc274198974b27c40c7e4e37 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
W20240507 08:12:15.119343 5654 leader_election.cc:341] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.123090 5654 leader_election.cc:341] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.123319 5654 leader_election.cc:341] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.123564 5654 leader_election.cc:341] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.123880 5654 consensus_peers.cc:485] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.125037 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_44bcd391ca2f4d08a04fd38c02b6f957"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.125537 7330 consensus_queue.cc:237] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.125696 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44bcd391ca2f4d08a04fd38c02b6f957 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.127054 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_43a808c00e4b4c99b2a63595d71acbd4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.127597 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43a808c00e4b4c99b2a63595d71acbd4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.128093 5546 tablet_service.cc:1467] Processing CreateTablet for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3 (DEFAULT_TABLE table=default.Table_a668c8aa1ac34ee799dac4150b2e59ec [id=d1cc3ddd685344628315c7e0024b82ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.128533 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 24e3ae77a7704423946b9074c19cb30d (DEFAULT_TABLE table=default.Table_73bd4c93636247cb834bbf7b46c39ad6 [id=861e7db1ad994b15bd71d3789e3885f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.128989 5548 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3151ab3214a1981f378903bb86a2d (DEFAULT_TABLE table=default.Table_6a42d94f854c4b529b014e54387d2869 [id=bd01d3e36ec94aaa9959bfc42f64a631]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.129894 5656 leader_election.cc:302] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.130303 5843 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.130503 5843 ts_tablet_manager.cc:1367] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.002s
I20240507 08:12:15.131332 7100 raft_consensus.cc:2781] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.131289 5843 raft_consensus.cc:348] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.131476 5843 raft_consensus.cc:374] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.131457 7100 raft_consensus.cc:686] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.131525 5843 raft_consensus.cc:729] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.131685 7100 consensus_queue.cc:237] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.131691 5843 consensus_queue.cc:260] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.131951 7394 consensus_queue.cc:1035] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.133170 7358 consensus_queue.cc:1035] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.133366 7333 consensus_queue.cc:1035] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.134184 5557 tablet_service.cc:1467] Processing CreateTablet for tablet fdffec6ce0da4c409466783bd76a6b4b (DEFAULT_TABLE table=default.Table_bbe582048e334931902cc697661690ca [id=d43b1125e24e4e79baffc5a279a81308]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.124125 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 14b073d71e934788a3b3575b52692d7b (DEFAULT_TABLE table=default.Table_381633ec2ac14aa4a5325fd26dafde7d [id=1c935e09f2f24adda618fc8b84093061]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.135466 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b073d71e934788a3b3575b52692d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.135663 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 1652772fff2d4ed9b50bb3022623eaa8 (DEFAULT_TABLE table=default.Table_5b0f0f6f249a4ceb83421dc56c43921e [id=34493ee6856149f283688904241d7df5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.134565 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 2b79d55827864491a6eddac99e383ba8 (DEFAULT_TABLE table=default.Table_f77c7b9058644cdd88fe3ee72b888872 [id=c770525101ca46c29545f045be2da1d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.136468 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.139289 5840 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:15.139412 5840 ts_tablet_manager.cc:1367] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.116s user 0.002s sys 0.001s
I20240507 08:12:15.140051 5840 raft_consensus.cc:348] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.140230 5840 raft_consensus.cc:374] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.140275 5840 raft_consensus.cc:729] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.140434 5840 consensus_queue.cc:260] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.140911 5840 ts_tablet_manager.cc:1398] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240507 08:12:15.140993 6513 ts_tablet_manager.cc:1875] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.141597 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3151ab3214a1981f378903bb86a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.141978 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b78c08d758043c78d5bb8281b7f2d62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.145821 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 695b2e6529914f51ba1884a216adbeb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.146026 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14b073d71e934788a3b3575b52692d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.146541 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e3ae77a7704423946b9074c19cb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.149776 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1652772fff2d4ed9b50bb3022623eaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.153837 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdffec6ce0da4c409466783bd76a6b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.157929 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79d55827864491a6eddac99e383ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.161780 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6a5953d1a3e4853bc3a7f21c46a6ee3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.162313 5563 tablet_service.cc:1467] Processing CreateTablet for tablet ad6bfed6bd8d4a56a990213caeaa4c35 (DEFAULT_TABLE table=default.Table_eecaf7ddbbe044fe86665d7b0fe5cfa2 [id=b0b76876d216431cb20d09f9508c9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.162767 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6bfed6bd8d4a56a990213caeaa4c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.165768 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24e3ae77a7704423946b9074c19cb30d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.166258 5840 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:15.167290 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdffec6ce0da4c409466783bd76a6b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.167745 5840 tablet_bootstrap.cc:654] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.169683 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3b065c19b8d486d99689f9d7c7b9c25. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.169675 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 6496a01b49b341a98e5be3e40d44e97e (DEFAULT_TABLE table=default.Table_e23516b85a2c422d8f09edaa41f0d52a [id=ce2f5c06cd4145ada11fd8129b35e3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.169998 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6496a01b49b341a98e5be3e40d44e97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.170890 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7dcf5643fd4c18ba7a69011421b5b8 (DEFAULT_TABLE table=default.Table_29d81546e07145b6aad46145e0f37793 [id=72b318bb15624f81a2c649ae37057b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.171180 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7dcf5643fd4c18ba7a69011421b5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.173686 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17276d2d36874841b0d41f0d1649a6b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.173992 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8 (DEFAULT_TABLE table=default.Table_47d427710f564bdebf6ad097dbfde129 [id=77f0b86c8b124a0095f281460627b569]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.174080 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9b0cde718548baad3ecca46e8000c9 (DEFAULT_TABLE table=default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.174324 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.177374 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9b0cde718548baad3ecca46e8000c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.180045 7101 raft_consensus.cc:482] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.180166 7101 raft_consensus.cc:504] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.180779 7101 leader_election.cc:288] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.180806 6513 ts_tablet_manager.cc:1888] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:15.180904 6513 log.cc:1196] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/79524df2415146ab926425c42068626c
I20240507 08:12:15.181183 6513 ts_tablet_manager.cc:1909] T 79524df2415146ab926425c42068626c P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:15.181865 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 6496a01b49b341a98e5be3e40d44e97e (DEFAULT_TABLE table=default.Table_e23516b85a2c422d8f09edaa41f0d52a [id=ce2f5c06cd4145ada11fd8129b35e3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.182197 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6496a01b49b341a98e5be3e40d44e97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.182343 5685 tablet_service.cc:1467] Processing CreateTablet for tablet ad6bfed6bd8d4a56a990213caeaa4c35 (DEFAULT_TABLE table=default.Table_eecaf7ddbbe044fe86665d7b0fe5cfa2 [id=b0b76876d216431cb20d09f9508c9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.182670 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6bfed6bd8d4a56a990213caeaa4c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.185940 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7dcf5643fd4c18ba7a69011421b5b8 (DEFAULT_TABLE table=default.Table_29d81546e07145b6aad46145e0f37793 [id=72b318bb15624f81a2c649ae37057b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.186242 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7dcf5643fd4c18ba7a69011421b5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.186937 5695 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3151ab3214a1981f378903bb86a2d (DEFAULT_TABLE table=default.Table_6a42d94f854c4b529b014e54387d2869 [id=bd01d3e36ec94aaa9959bfc42f64a631]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.187227 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3151ab3214a1981f378903bb86a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.190984 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.191494 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8 (DEFAULT_TABLE table=default.Table_47d427710f564bdebf6ad097dbfde129 [id=77f0b86c8b124a0095f281460627b569]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.191828 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.192432 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b79d55827864491a6eddac99e383ba8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.138640 5843 ts_tablet_manager.cc:1398] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:12:15.159266 5416 tablet_service.cc:1467] Processing CreateTablet for tablet c2a3151ab3214a1981f378903bb86a2d (DEFAULT_TABLE table=default.Table_6a42d94f854c4b529b014e54387d2869 [id=bd01d3e36ec94aaa9959bfc42f64a631]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.193938 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a3151ab3214a1981f378903bb86a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.194365 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1652772fff2d4ed9b50bb3022623eaa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.162259 5425 tablet_service.cc:1467] Processing CreateTablet for tablet ad6bfed6bd8d4a56a990213caeaa4c35 (DEFAULT_TABLE table=default.Table_eecaf7ddbbe044fe86665d7b0fe5cfa2 [id=b0b76876d216431cb20d09f9508c9484]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.197870 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad6bfed6bd8d4a56a990213caeaa4c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.199247 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cc798eb4e02c456880ef3014ebbb7e24
I20240507 08:12:15.169847 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 6496a01b49b341a98e5be3e40d44e97e (DEFAULT_TABLE table=default.Table_e23516b85a2c422d8f09edaa41f0d52a [id=ce2f5c06cd4145ada11fd8129b35e3ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.200071 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6496a01b49b341a98e5be3e40d44e97e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.200562 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.200762 5445 raft_consensus.cc:2454] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.202833 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.203503 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2dc142f4782144e49dfe1eb829d61be8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.204090 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2dc142f4782144e49dfe1eb829d61be8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.206172 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.171272 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8b7dcf5643fd4c18ba7a69011421b5b8 (DEFAULT_TABLE table=default.Table_29d81546e07145b6aad46145e0f37793 [id=72b318bb15624f81a2c649ae37057b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.208138 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b7dcf5643fd4c18ba7a69011421b5b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.210486 5654 leader_election.cc:341] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.210690 5654 consensus_peers.cc:485] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:15.212170 5654 consensus_peers.cc:485] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.212404 7101 consensus_queue.cc:1035] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.213042 5656 leader_election.cc:302] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.213305 7101 raft_consensus.cc:2781] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.213375 7101 raft_consensus.cc:482] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.213428 7101 raft_consensus.cc:3037] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.220137 5843 tablet_bootstrap.cc:492] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.192071 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8 (DEFAULT_TABLE table=default.Table_47d427710f564bdebf6ad097dbfde129 [id=77f0b86c8b124a0095f281460627b569]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.226203 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fa2f5cfb3f144d095fbd117baf9d8c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.226686 7101 raft_consensus.cc:504] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.227311 7101 leader_election.cc:288] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.227944 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet cc798eb4e02c456880ef3014ebbb7e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.228328 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f748727d3a88442d90cf81bae48acd36
I20240507 08:12:15.228829 5258 catalog_manager.cc:6554] ReplaceTablet: tablet f748727d3a88442d90cf81bae48acd36 deleted and replaced by tablet fe3beea5625b43259d0635a7ee1ef82b
I20240507 08:12:15.229887 5843 tablet_bootstrap.cc:654] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.230197 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.230394 7101 consensus_queue.cc:1035] T b9e715a29b0643b6ac61f9e73874c25c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.230517 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f3bdb777b974524b5673a3f02883107" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.230688 5444 raft_consensus.cc:3037] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:15.230720 5654 leader_election.cc:341] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.231846 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet f748727d3a88442d90cf81bae48acd36 with delete_type TABLET_DATA_DELETED (replaced by tablet fe3beea5625b43259d0635a7ee1ef82b) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.231873 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet f748727d3a88442d90cf81bae48acd36 with delete_type TABLET_DATA_DELETED (replaced by tablet fe3beea5625b43259d0635a7ee1ef82b) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.232753 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6b76dffef8004978b959721044683956" } modify_external_catalogs: true
I20240507 08:12:15.232966 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6b76dffef8004978b959721044683956" } modify_external_catalogs: true
I20240507 08:12:15.233186 5444 raft_consensus.cc:2454] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:15.234302 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet f748727d3a88442d90cf81bae48acd36 with delete_type TABLET_DATA_DELETED (replaced by tablet fe3beea5625b43259d0635a7ee1ef82b) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.236732 5656 leader_election.cc:302] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.237296 7101 raft_consensus.cc:2781] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.237452 7101 raft_consensus.cc:686] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.237830 7101 consensus_queue.cc:237] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.242895 5841 ts_tablet_manager.cc:1888] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.243055 5841 log.cc:1196] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:12:15.243463 5841 ts_tablet_manager.cc:1909] T b27cc38d0c4c4083900cd6765a5cf6e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.243790 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1022) took 3593 ms. Trace:
I20240507 08:12:15.243913 5841 rpcz_store.cc:276] 0507 08:12:11.650164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.650396 (+ 232us) service_pool.cc:225] Handling call
0507 08:12:15.065554 (+3415158us) ts_tablet_manager.cc:1120] Deleting tablet b27cc38d0c4c4083900cd6765a5cf6e6
0507 08:12:15.065562 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.084429 (+ 18867us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.242741 (+158312us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.243772 (+ 1031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3414987}
I20240507 08:12:15.244498 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b27cc38d0c4c4083900cd6765a5cf6e6 (table default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]) successfully deleted
I20240507 08:12:15.244611 5841 tablet_replica.cc:330] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.244794 5841 raft_consensus.cc:2229] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.244961 5841 raft_consensus.cc:2258] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.245357 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
W20240507 08:12:15.246320 5654 consensus_peers.cc:485] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.253484 5416 tablet_service.cc:1467] Processing CreateTablet for tablet eb1009eeaf86405b8e082e66ead9fc11 (DEFAULT_TABLE table=default.Table_5bbebe0430784c14affcd57084473b5a [id=c1046dce493b4063828e8d8200f33486]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.253993 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1009eeaf86405b8e082e66ead9fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.254026 5546 tablet_service.cc:1467] Processing CreateTablet for tablet eb1009eeaf86405b8e082e66ead9fc11 (DEFAULT_TABLE table=default.Table_5bbebe0430784c14affcd57084473b5a [id=c1046dce493b4063828e8d8200f33486]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.254478 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1009eeaf86405b8e082e66ead9fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.260349 7408 raft_consensus.cc:2781] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.260452 7408 raft_consensus.cc:482] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.260519 7408 raft_consensus.cc:3037] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.263278 7408 raft_consensus.cc:504] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.264215 7408 leader_election.cc:288] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.265103 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a015bf84a454d49b6497c16ed15c84a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:15.265763 5378 leader_election.cc:341] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.267098 5843 tablet_bootstrap.cc:492] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.267283 5843 ts_tablet_manager.cc:1367] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.002s
I20240507 08:12:15.267494 5579 raft_consensus.cc:3037] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.268007 5843 raft_consensus.cc:348] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.268178 5843 raft_consensus.cc:374] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.268235 5843 raft_consensus.cc:729] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.268445 5843 consensus_queue.cc:260] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.269084 5843 ts_tablet_manager.cc:1398] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:15.269522 5843 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.271009 5843 tablet_bootstrap.cc:654] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.272589 5552 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5ba0221304453a00bb779aee676d4 (DEFAULT_TABLE table=default.Table_ea12b1a814f546a19315986dde83e3db [id=caff73c6cabf4eefaf6564270288143a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.273037 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5ba0221304453a00bb779aee676d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.276165 5683 tablet_service.cc:1467] Processing CreateTablet for tablet eb1009eeaf86405b8e082e66ead9fc11 (DEFAULT_TABLE table=default.Table_5bbebe0430784c14affcd57084473b5a [id=c1046dce493b4063828e8d8200f33486]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.276629 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1009eeaf86405b8e082e66ead9fc11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.278352 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1000) took 4566 ms. Trace:
I20240507 08:12:15.278482 6513 rpcz_store.cc:276] 0507 08:12:10.712247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.712305 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:15.043056 (+4330751us) ts_tablet_manager.cc:1120] Deleting tablet 79524df2415146ab926425c42068626c
0507 08:12:15.043223 (+ 167us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.154470 (+111247us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.180727 (+ 26257us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.278332 (+ 97605us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4330594}
I20240507 08:12:15.278884 6513 tablet_replica.cc:330] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:15.279111 6513 raft_consensus.cc:2229] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:15.279436 6513 raft_consensus.cc:2258] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.280016 6513 ts_tablet_manager.cc:1875] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.281749 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 79524df2415146ab926425c42068626c (table default.Table_16b866688308450cb4d4f81ccafa1afb [id=35093b3de3164b9c809aae8b5da3db54]) successfully deleted
I20240507 08:12:15.282130 5681 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5ba0221304453a00bb779aee676d4 (DEFAULT_TABLE table=default.Table_ea12b1a814f546a19315986dde83e3db [id=caff73c6cabf4eefaf6564270288143a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.282454 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5ba0221304453a00bb779aee676d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.282460 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6837510d0460f9e5ef041380b35a7 (DEFAULT_TABLE table=default.Table_1e1cfba5e8234af8823ff12bdefb9723 [id=de9df45f901348d9abf1aeb49d933764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.282804 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6837510d0460f9e5ef041380b35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.293820 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9b0cde718548baad3ecca46e8000c9 (DEFAULT_TABLE table=default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.295938 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6837510d0460f9e5ef041380b35a7 (DEFAULT_TABLE table=default.Table_1e1cfba5e8234af8823ff12bdefb9723 [id=de9df45f901348d9abf1aeb49d933764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.298485 5556 tablet_service.cc:1467] Processing CreateTablet for tablet e87ee005f2694359b74c290365eb0fb7 (DEFAULT_TABLE table=default.Table_42cf3e4b98c64c0db11123bb16206ebf [id=583a7999db274c42ac6a49853c3f22b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.298920 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87ee005f2694359b74c290365eb0fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.301680 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9b0cde718548baad3ecca46e8000c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.306766 5552 tablet_service.cc:1467] Processing CreateTablet for tablet e888306f97d445a580d4faf2cb050f0b (DEFAULT_TABLE table=default.Table_8b032f4a53674fa3a9bcb78cd817e5e3 [id=ed547fdd3c9141ff96cb34c242559f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.307205 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888306f97d445a580d4faf2cb050f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.307571 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6837510d0460f9e5ef041380b35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.311872 5421 tablet_service.cc:1467] Processing CreateTablet for tablet e888306f97d445a580d4faf2cb050f0b (DEFAULT_TABLE table=default.Table_8b032f4a53674fa3a9bcb78cd817e5e3 [id=ed547fdd3c9141ff96cb34c242559f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.312255 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888306f97d445a580d4faf2cb050f0b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.314184 5654 consensus_peers.cc:485] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.314694 5698 tablet_service.cc:1467] Processing CreateTablet for tablet e888306f97d445a580d4faf2cb050f0b (DEFAULT_TABLE table=default.Table_8b032f4a53674fa3a9bcb78cd817e5e3 [id=ed547fdd3c9141ff96cb34c242559f7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.315062 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e888306f97d445a580d4faf2cb050f0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.272728 5413 tablet_service.cc:1467] Processing CreateTablet for tablet 4a9b0cde718548baad3ecca46e8000c9 (DEFAULT_TABLE table=default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.315889 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a9b0cde718548baad3ecca46e8000c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.317667 5560 tablet_service.cc:1467] Processing CreateTablet for tablet ffc89e00efaa4bde9242be40cf409c59 (DEFAULT_TABLE table=default.Table_77bee6f5db85437bb6ecccd959ba929f [id=a5ff5199108542cea9572a0582fffa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.318006 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc89e00efaa4bde9242be40cf409c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.321271 7378 raft_consensus.cc:482] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.321403 7378 raft_consensus.cc:504] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.322005 7378 leader_election.cc:288] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.322566 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c6f552832a45c4a2e28f40797db8fa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.314386 5683 tablet_service.cc:1467] Processing CreateTablet for tablet e87ee005f2694359b74c290365eb0fb7 (DEFAULT_TABLE table=default.Table_42cf3e4b98c64c0db11123bb16206ebf [id=583a7999db274c42ac6a49853c3f22b4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.322996 5378 leader_election.cc:341] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.323145 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87ee005f2694359b74c290365eb0fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.325556 5843 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.325707 5843 ts_tablet_manager.cc:1367] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.002s
W20240507 08:12:15.326414 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648113d9b8164cc59ba238ace8f65b39 consensus state has no leader
I20240507 08:12:15.329779 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]). Expected version 2 got 0
I20240507 08:12:15.330507 5262 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_166bf386f4e046d7a019c2b3a2689225 [id=83f625713cda401f90e513ddef199bce] alter complete (version 1)
W20240507 08:12:15.330971 5262 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.194:35513 (request call id 209) took 779 ms (client timeout 1000 ms). Trace:
I20240507 08:12:15.331290 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet d451d03349794726a71c255a7f177436 with delete_type TABLET_DATA_DELETED (replaced by tablet 1227b63fbc6b4625bef0b37404b76a6f) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:15.331073 5262 rpcz_store.cc:269] 0507 08:12:14.551442 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.569967 (+ 18525us) service_pool.cc:225] Handling call
0507 08:12:14.578038 (+ 8071us) spinlock_profiling.cc:243] Waited 7.51 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9bf668 00007f3bc39b3786 00007f3bc39b34b5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:15.330897 (+752859us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:14.640810 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:14.640966 (+ 156us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:14.640972 (+ 6us) write_op.cc:435] Acquired schema lock
0507 08:12:14.640974 (+ 2us) tablet.cc:660] Decoding operations
0507 08:12:14.641089 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:14.641101 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:14.641106 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0507 08:12:14.641167 (+ 61us) tablet.cc:699] Row locks acquired
0507 08:12:14.641169 (+ 2us) write_op.cc:260] PREPARE: finished
0507 08:12:14.641208 (+ 39us) write_op.cc:270] Start()
0507 08:12:14.641236 (+ 28us) write_op.cc:276] Timestamp: P: 1715069534641152 usec, L: 0
0507 08:12:14.641240 (+ 4us) op_driver.cc:331] REPLICATION: starting
0507 08:12:14.641423 (+ 183us) log.cc:842] Serialized 356 byte log entry
0507 08:12:14.644031 (+ 2608us) op_driver.cc:444] REPLICATION: finished
0507 08:12:14.644193 (+ 162us) write_op.cc:301] APPLY: starting
0507 08:12:14.644215 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:14.644250 (+ 35us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:14.644274 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:14.644319 (+ 45us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:14.644348 (+ 29us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0507 08:12:14.644355 (+ 7us) write_op.cc:312] APPLY: finished
0507 08:12:14.644412 (+ 57us) log.cc:842] Serialized 23 byte log entry
0507 08:12:14.644457 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:14.644472 (+ 15us) write_op.cc:454] Released schema lock
0507 08:12:14.644478 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:14.644552 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:14.644555 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":17278720,"child_traces":[["op",{"apply.queue_time_us":126,"apply.run_cpu_time_us":399,"apply.run_wall_time_us":400,"num_ops":1,"prepare.queue_time_us":60373,"prepare.run_cpu_time_us":709,"prepare.run_wall_time_us":10652,"raft.queue_time_us":30,"raft.run_cpu_time_us":204,"raft.run_wall_time_us":204,"replication_time_us":2770}]]}
I20240507 08:12:15.331871 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 453618a1fb9b415ca06a4567c63d4a44 (DEFAULT_TABLE table=default.Table_8d1d33ea89b74c3da59e90ff6253f4e0 [id=6d9bb0f13a9a4d9185deacda57f46661]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.332283 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453618a1fb9b415ca06a4567c63d4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.333336 5698 tablet_service.cc:1467] Processing CreateTablet for tablet ffc89e00efaa4bde9242be40cf409c59 (DEFAULT_TABLE table=default.Table_77bee6f5db85437bb6ecccd959ba929f [id=a5ff5199108542cea9572a0582fffa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.333721 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc89e00efaa4bde9242be40cf409c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.334553 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1037) took 2461 ms. Trace:
W20240507 08:12:15.335373 5259 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.193:55019 (request call id 159) took 992 ms (client timeout 1000 ms). Trace:
W20240507 08:12:15.335490 5259 rpcz_store.cc:269] 0507 08:12:14.342917 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.352395 (+ 9478us) service_pool.cc:225] Handling call
0507 08:12:14.358110 (+ 5715us) spinlock_profiling.cc:243] Waited 4.96 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9bf668 00007f3bc39b3786 00007f3bc39b34b5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:15.335306 (+977196us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":11397376}
I20240507 08:12:15.337416 5843 raft_consensus.cc:348] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.339567 7079 rpcz_store.cc:276] 0507 08:12:12.872768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.872819 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:14.828349 (+1955530us) ts_tablet_manager.cc:1120] Deleting tablet 1037c3c0af5e464bb23a311e552fcf21
0507 08:12:14.828357 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:14.897746 (+ 69389us) spinlock_profiling.cc:243] Waited 68 ms on lock 0x616000172654. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2af3ca00 00007f2f2c216b83 00007f2f2ecaa4ec 00007f2f2ecdc844 00007f2f2ecbdb30 00007f2f2ecbd8bd 0000000000a60bcc 00007f2f1f714b74 00007f2f1f71819b 00007f2f1f718018
0507 08:12:15.012528 (+114782us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.015411 (+ 2883us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.334534 (+319123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":156451072,"tablet-delete.queue_time_us":1955344}
I20240507 08:12:15.340220 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1037c3c0af5e464bb23a311e552fcf21 (table default.Table_64af22be42cd4e0d84bc79cfcdba4f20 [id=20f9b0b0ba424075871ca842c353b8e4]) successfully deleted
I20240507 08:12:15.281152 5417 tablet_service.cc:1467] Processing CreateTablet for tablet a0f5ba0221304453a00bb779aee676d4 (DEFAULT_TABLE table=default.Table_ea12b1a814f546a19315986dde83e3db [id=caff73c6cabf4eefaf6564270288143a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.341313 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0f5ba0221304453a00bb779aee676d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.295629 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 78b6837510d0460f9e5ef041380b35a7 (DEFAULT_TABLE table=default.Table_1e1cfba5e8234af8823ff12bdefb9723 [id=de9df45f901348d9abf1aeb49d933764]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.344267 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78b6837510d0460f9e5ef041380b35a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.280059 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 193588us
I20240507 08:12:15.303977 5425 tablet_service.cc:1467] Processing CreateTablet for tablet e87ee005f2694359b74c290365eb0fb7 (DEFAULT_TABLE table=default.Table_42cf3e4b98c64c0db11123bb16206ebf [id=583a7999db274c42ac6a49853c3f22b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.338243 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.349951 7079 tablet_replica.cc:330] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:15.350239 7079 raft_consensus.cc:2229] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.350584 7079 raft_consensus.cc:2258] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.338369 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.352417 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c6f552832a45c4a2e28f40797db8fa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.352644 5720 raft_consensus.cc:2454] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:15.338721 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 453618a1fb9b415ca06a4567c63d4a44 (DEFAULT_TABLE table=default.Table_8d1d33ea89b74c3da59e90ff6253f4e0 [id=6d9bb0f13a9a4d9185deacda57f46661]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.353922 7079 ts_tablet_manager.cc:1875] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.355100 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e87ee005f2694359b74c290365eb0fb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.367184 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5844bcf059ea43c6a4e0c092a9f8d3d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.367781 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5844bcf059ea43c6a4e0c092a9f8d3d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.369287 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_74382721d3fd4f3dac0a931308d156a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.369813 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74382721d3fd4f3dac0a931308d156a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.369858 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453618a1fb9b415ca06a4567c63d4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.371752 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4d223c1ac6e944a4bff2404a133840dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.372192 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d223c1ac6e944a4bff2404a133840dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.351856 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a015bf84a454d49b6497c16ed15c84a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:15.378444 5718 raft_consensus.cc:3037] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.296310 5841 ts_tablet_manager.cc:1875] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.381261 5718 raft_consensus.cc:2454] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:15.383059 5416 tablet_service.cc:1467] Processing CreateTablet for tablet ffc89e00efaa4bde9242be40cf409c59 (DEFAULT_TABLE table=default.Table_77bee6f5db85437bb6ecccd959ba929f [id=a5ff5199108542cea9572a0582fffa9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.388082 7101 consensus_queue.cc:1035] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.390916 5840 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:15.391124 5840 ts_tablet_manager.cc:1367] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.225s user 0.006s sys 0.003s
I20240507 08:12:15.391312 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc89e00efaa4bde9242be40cf409c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.391983 5840 raft_consensus.cc:348] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.392237 5840 raft_consensus.cc:374] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.392343 5840 raft_consensus.cc:729] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.392580 5840 consensus_queue.cc:260] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.393100 5840 ts_tablet_manager.cc:1398] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:15.393487 5840 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:15.397729 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c649ec5b00df4a4e851f765549a0d9f8" } new_table_name: "default.Table_a3ed2876b6214485a0f2bbfb0d196d5a" modify_external_catalogs: true
I20240507 08:12:15.399581 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:15.400354 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_a63ec02af6ab4e85830582d44267f8c1 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 1 got 0
I20240507 08:12:15.400856 7378 raft_consensus.cc:482] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.400983 7378 raft_consensus.cc:504] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.401558 7378 leader_election.cc:288] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.409727 5843 raft_consensus.cc:374] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.409838 5843 raft_consensus.cc:729] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.409968 5255 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet fe3beea5625b43259d0635a7ee1ef82b
I20240507 08:12:15.410069 5843 consensus_queue.cc:260] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.410279 5656 consensus_peers.cc:485] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 957775c01be94cc081db344c95a81310. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.410681 5843 ts_tablet_manager.cc:1398] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.085s user 0.002s sys 0.000s
I20240507 08:12:15.411048 5843 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.411688 5377 leader_election.cc:302] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.412086 7378 raft_consensus.cc:2781] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.412163 7378 raft_consensus.cc:482] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.412220 7378 raft_consensus.cc:3037] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.414682 7330 consensus_queue.cc:1035] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.414779 7378 raft_consensus.cc:504] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.415634 7378 leader_election.cc:288] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.416113 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c6f552832a45c4a2e28f40797db8fa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:15.416574 5378 leader_election.cc:341] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.417069 5377 leader_election.cc:302] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.417385 7408 raft_consensus.cc:2781] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.417491 7408 raft_consensus.cc:686] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:15.417729 7408 consensus_queue.cc:237] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:15.418929 5656 consensus_peers.cc:485] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31251576beee4a8a81a453b721c9c058. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.422011 5843 tablet_bootstrap.cc:654] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.422029 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14929efb0ed049d6b554598a5b8cb494" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.422223 5718 raft_consensus.cc:2454] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:15.431473 5840 tablet_bootstrap.cc:654] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.432940 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8c6f552832a45c4a2e28f40797db8fa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:15.433126 5718 raft_consensus.cc:3037] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.435690 5718 raft_consensus.cc:2454] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:15.440203 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14929efb0ed049d6b554598a5b8cb494" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:15.440753 5378 leader_election.cc:341] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.441123 5841 ts_tablet_manager.cc:1888] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.442412 5841 log.cc:1196] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:15.442844 5841 ts_tablet_manager.cc:1909] T 5471c74c33cd42e8aaf47f71f61adff7 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.443691 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1020) took 3793 ms. Trace:
W20240507 08:12:15.444731 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.365s user 0.002s sys 0.000s
I20240507 08:12:15.445057 5841 rpcz_store.cc:276] 0507 08:12:11.649967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.650858 (+ 891us) service_pool.cc:225] Handling call
0507 08:12:15.244538 (+3593680us) ts_tablet_manager.cc:1120] Deleting tablet 5471c74c33cd42e8aaf47f71f61adff7
0507 08:12:15.244548 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.389751 (+145203us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.433383 (+ 43632us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.443671 (+ 10288us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3593489}
I20240507 08:12:15.446292 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5471c74c33cd42e8aaf47f71f61adff7 (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:12:15.446887 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1024) took 3748 ms. Trace:
I20240507 08:12:15.447894 5841 rpcz_store.cc:276] 0507 08:12:11.698142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.698318 (+ 176us) service_pool.cc:225] Handling call
0507 08:12:15.446747 (+3748429us) ts_tablet_manager.cc:1120] Deleting tablet 46cc8597ca5042ca8e85d99803d07206
0507 08:12:15.446759 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.446870 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3748218}
W20240507 08:12:15.448879 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 46cc8597ca5042ca8e85d99803d07206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cc8597ca5042ca8e85d99803d07206 already in progress: creating tablet
I20240507 08:12:15.449525 5377 leader_election.cc:302] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.449944 7378 raft_consensus.cc:2781] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.450021 7378 raft_consensus.cc:482] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.450098 7378 raft_consensus.cc:3037] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.450305 5841 tablet_replica.cc:330] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.452687 7378 raft_consensus.cc:504] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.454560 7378 leader_election.cc:288] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.456744 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14929efb0ed049d6b554598a5b8cb494" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:15.457222 5378 leader_election.cc:341] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.457268 5377 leader_election.cc:302] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.457715 5841 raft_consensus.cc:2229] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.459407 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14929efb0ed049d6b554598a5b8cb494" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:15.459599 5718 raft_consensus.cc:3037] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.462499 5718 raft_consensus.cc:2454] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:15.457973 7378 raft_consensus.cc:2781] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.463526 7378 raft_consensus.cc:686] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:15.463783 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8f670d8a09204a8fb9319c3d39bd175f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.463812 7378 consensus_queue.cc:237] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.464434 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f670d8a09204a8fb9319c3d39bd175f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.465981 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_15f91f4b2cc24f138a6758b2a182bee7" } new_table_name: "default.Table_e2323a288cf147008fb6c7f1273d8497" modify_external_catalogs: true
I20240507 08:12:15.467262 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_378c906ca8ff4cf199a515389bd939d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.467908 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_378c906ca8ff4cf199a515389bd939d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.472492 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 453618a1fb9b415ca06a4567c63d4a44 (DEFAULT_TABLE table=default.Table_8d1d33ea89b74c3da59e90ff6253f4e0 [id=6d9bb0f13a9a4d9185deacda57f46661]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.475716 5377 leader_election.cc:302] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.476225 7408 raft_consensus.cc:2781] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.481062 5841 raft_consensus.cc:2258] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.482142 5841 ts_tablet_manager.cc:1875] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.463348 5843 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.485646 5843 ts_tablet_manager.cc:1367] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.004s
I20240507 08:12:15.486403 5843 raft_consensus.cc:348] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.486672 5843 raft_consensus.cc:374] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.486775 5843 raft_consensus.cc:729] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.487047 5843 consensus_queue.cc:260] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.487676 5843 ts_tablet_manager.cc:1398] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:15.490262 7408 raft_consensus.cc:686] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:15.490723 7408 consensus_queue.cc:237] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.494801 5843 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.495693 6675 consensus_queue.cc:1035] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.496176 6513 ts_tablet_manager.cc:1888] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.496881 5843 tablet_bootstrap.cc:654] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.499096 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c707458d1c094d87b2204683b28b49e4
I20240507 08:12:15.500859 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet c707458d1c094d87b2204683b28b49e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:15.503302 5261 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.4.246.195:47089 (request call id 130) took 1067 ms (client timeout 1000 ms). Trace:
W20240507 08:12:15.503437 5261 rpcz_store.cc:269] 0507 08:12:14.436048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.441984 (+ 5936us) service_pool.cc:225] Handling call
0507 08:12:14.451998 (+ 10014us) spinlock_profiling.cc:243] Waited 7.98 ms on lock 0x616000021d54. stack: 00007f3bb93da756 00007f3bb93da510 00007f3bb840d48d 0000000000955804 00000000009557cd 00007f3bc7f1a15d 00007f3bc7f1a253 00007f3bc4c4bd41 00007f3bcc83834d 00007f3bcc9bf668 00007f3bc39b3786 00007f3bc39b34b5 00007f3bc00e204e 00007f3bc00e0f2e 00007f3bc00e7405 00007f3bc00e7d2f
0507 08:12:15.503256 (+1051258us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0507 08:12:15.233706 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0507 08:12:15.233863 (+ 157us) write_op.cc:432] Acquiring schema lock in shared mode
0507 08:12:15.233872 (+ 9us) write_op.cc:435] Acquired schema lock
0507 08:12:15.233876 (+ 4us) tablet.cc:660] Decoding operations
0507 08:12:15.234111 (+ 235us) write_op.cc:620] Acquiring the partition lock for write op
0507 08:12:15.234130 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0507 08:12:15.234137 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0507 08:12:15.260521 (+ 26384us) tablet.cc:699] Row locks acquired
0507 08:12:15.260528 (+ 7us) write_op.cc:260] PREPARE: finished
0507 08:12:15.260590 (+ 62us) write_op.cc:270] Start()
0507 08:12:15.261600 (+ 1010us) write_op.cc:276] Timestamp: P: 1715069535260530 usec, L: 0
0507 08:12:15.261608 (+ 8us) op_driver.cc:331] REPLICATION: starting
0507 08:12:15.261901 (+ 293us) log.cc:842] Serialized 1674 byte log entry
0507 08:12:15.324618 (+ 62717us) op_driver.cc:444] REPLICATION: finished
0507 08:12:15.325134 (+ 516us) write_op.cc:301] APPLY: starting
0507 08:12:15.325153 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0507 08:12:15.325215 (+ 62us) tablet.cc:1368] finished BulkCheckPresence
0507 08:12:15.325259 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0507 08:12:15.325348 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0507 08:12:15.325384 (+ 36us) tablet_metrics.cc:523] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0507 08:12:15.325391 (+ 7us) write_op.cc:312] APPLY: finished
0507 08:12:15.325481 (+ 90us) log.cc:842] Serialized 41 byte log entry
0507 08:12:15.325542 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:15.325565 (+ 23us) write_op.cc:454] Released schema lock
0507 08:12:15.325593 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0507 08:12:15.325661 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0507 08:12:15.325665 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":46,"spinlock_wait_cycles":18355072,"child_traces":[["op",{"apply.queue_time_us":459,"apply.run_cpu_time_us":587,"apply.run_wall_time_us":586,"num_ops":4,"prepare.queue_time_us":198097,"prepare.run_cpu_time_us":1273,"prepare.run_wall_time_us":52660,"raft.queue_time_us":46,"raft.run_cpu_time_us":270,"raft.run_wall_time_us":271,"replication_time_us":62950}]]}
I20240507 08:12:15.505589 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ee96356f684941cc995051508593ecd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.506172 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee96356f684941cc995051508593ecd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.515234 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ff9a2c687f9446dab1f59c2df5d949bf" } new_table_name: "default.Table_31ce9bd4d55e4973aabbcd6d73982530" modify_external_catalogs: true
I20240507 08:12:15.519665 6513 log.cc:1196] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b6e68364238c4b66b8c0de0f4a2ea855
I20240507 08:12:15.522184 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.502720 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 453618a1fb9b415ca06a4567c63d4a44. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.522596 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.522825 6513 ts_tablet_manager.cc:1909] T b6e68364238c4b66b8c0de0f4a2ea855 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:15.523923 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1001) took 4811 ms. Trace:
I20240507 08:12:15.514935 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet c707458d1c094d87b2204683b28b49e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:15.534075 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be1b36ca5b6041d780e851f137d3a4c9 consensus state has no leader
I20240507 08:12:15.534241 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.534395 7378 raft_consensus.cc:482] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.534570 7378 raft_consensus.cc:504] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.535671 7378 leader_election.cc:288] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.535981 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.536357 5841 ts_tablet_manager.cc:1888] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.536473 5841 log.cc:1196] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/cd4379b6fded4a52843f55f27ae67121
W20240507 08:12:15.536623 5378 leader_election.cc:341] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.536824 5841 ts_tablet_manager.cc:1909] T cd4379b6fded4a52843f55f27ae67121 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.537158 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1025) took 3838 ms. Trace:
I20240507 08:12:15.537199 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8493d14dd0f8480591e6e0a0196c77cb" } modify_external_catalogs: true
I20240507 08:12:15.537425 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8493d14dd0f8480591e6e0a0196c77cb" } modify_external_catalogs: true
I20240507 08:12:15.537284 5841 rpcz_store.cc:276] 0507 08:12:11.698263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.698328 (+ 65us) service_pool.cc:225] Handling call
0507 08:12:15.450235 (+3751907us) ts_tablet_manager.cc:1120] Deleting tablet cd4379b6fded4a52843f55f27ae67121
0507 08:12:15.450245 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.509773 (+ 59528us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.536221 (+ 26448us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.537138 (+ 917us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3751680}
I20240507 08:12:15.538879 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_02c584c9b6774d37b485b35ba37050ec" } new_table_name: "default.Table_f0a52c2d56e7410cbdfd1b5b539f4bc2" modify_external_catalogs: true
I20240507 08:12:15.531885 6513 rpcz_store.cc:276] 0507 08:12:10.712604 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.712646 (+ 42us) service_pool.cc:225] Handling call
0507 08:12:15.278844 (+4566198us) ts_tablet_manager.cc:1120] Deleting tablet b6e68364238c4b66b8c0de0f4a2ea855
0507 08:12:15.278853 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.391751 (+112898us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.496070 (+104319us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.523906 (+ 27836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4566040}
I20240507 08:12:15.540283 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1002) took 4827 ms. Trace:
I20240507 08:12:15.540362 6513 rpcz_store.cc:276] 0507 08:12:10.713021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.713065 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:15.540173 (+4827108us) ts_tablet_manager.cc:1120] Deleting tablet 070fdbf0b00b4582a184729c2f2a4b69
0507 08:12:15.540185 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.540264 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4826970}
I20240507 08:12:15.542789 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet cd4379b6fded4a52843f55f27ae67121 (table default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]) successfully deleted
I20240507 08:12:15.542866 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet c707458d1c094d87b2204683b28b49e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.543394 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1026) took 3841 ms. Trace:
I20240507 08:12:15.515053 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.516954 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.517743 5840 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:15.546448 5840 ts_tablet_manager.cc:1367] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.153s user 0.003s sys 0.006s
I20240507 08:12:15.514883 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c7c39ec9fec4331afabb39d1483ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:53 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.547377 5841 rpcz_store.cc:276] 0507 08:12:11.702326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.702395 (+ 69us) service_pool.cc:225] Handling call
0507 08:12:15.542811 (+3840416us) ts_tablet_manager.cc:1120] Deleting tablet d4fac82e12d0437e950d01dbc4117efa
0507 08:12:15.542821 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.543369 (+ 548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3840239}
I20240507 08:12:15.548424 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1003) took 4816 ms. Trace:
I20240507 08:12:15.548580 6513 rpcz_store.cc:276] 0507 08:12:10.723937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:10.723985 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:15.540576 (+4816591us) ts_tablet_manager.cc:1120] Deleting tablet 5c49745fce2949d1909baf7583b157a7
0507 08:12:15.540585 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.540679 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4816424}
I20240507 08:12:15.549556 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 35e34b60b78b402faf51e003c97db787 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.549840 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1004) took 4451 ms. Trace:
I20240507 08:12:15.549961 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de26190bc44916809856d84ad68345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.550341 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.551268 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet 44ec0526e1914221bc22b3e75ea01e0d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.556185 5841 tablet_replica.cc:330] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.556403 5841 raft_consensus.cc:2229] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.556552 5841 raft_consensus.cc:2258] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.556913 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_948f06376d4849c79cc3f222a3fcaaad" } modify_external_catalogs: true
I20240507 08:12:15.557134 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_948f06376d4849c79cc3f222a3fcaaad" } modify_external_catalogs: true
I20240507 08:12:15.558920 7100 raft_consensus.cc:482] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.559093 7100 raft_consensus.cc:504] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.559785 7100 leader_election.cc:288] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.549962 6513 rpcz_store.cc:276] 0507 08:12:11.098571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.098648 (+ 77us) service_pool.cc:225] Handling call
0507 08:12:15.548907 (+4450259us) ts_tablet_manager.cc:1120] Deleting tablet 0c51ec8f693e4f108100c75470ead2a9
0507 08:12:15.548918 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.549825 (+ 907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4450037}
I20240507 08:12:15.561272 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1005) took 4457 ms. Trace:
I20240507 08:12:15.561355 6513 rpcz_store.cc:276] 0507 08:12:11.103702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.104183 (+ 481us) service_pool.cc:225] Handling call
0507 08:12:15.561161 (+4456978us) ts_tablet_manager.cc:1120] Deleting tablet 7fbe0f58bad541e7bcfb611105328d56
0507 08:12:15.561170 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.561256 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4456751}
I20240507 08:12:15.561633 6513 tablet_replica.cc:330] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:15.561815 6513 raft_consensus.cc:2229] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:15.562336 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.562582 5446 raft_consensus.cc:2454] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.562969 5656 leader_election.cc:302] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:15.570174 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1319db1a6ea41f1a4eecf02ff67d217 consensus state has no leader
I20240507 08:12:15.571784 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9196f7dce1f74149b05083887b83f5b0" } modify_external_catalogs: true
I20240507 08:12:15.571985 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9196f7dce1f74149b05083887b83f5b0" } modify_external_catalogs: true
I20240507 08:12:15.575250 5839 tablet_bootstrap.cc:492] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:15.575477 5839 ts_tablet_manager.cc:1367] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.539s user 0.006s sys 0.026s
I20240507 08:12:15.576723 5839 raft_consensus.cc:348] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.577028 5839 raft_consensus.cc:374] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.577131 5839 raft_consensus.cc:729] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.577446 5839 consensus_queue.cc:260] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.579942 5839 ts_tablet_manager.cc:1398] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:15.581988 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:15.585155 5841 ts_tablet_manager.cc:1875] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:15.593428 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0b47825cbd714b91b729118e5f01eb16 consensus state has no leader
I20240507 08:12:15.593379 5840 raft_consensus.cc:348] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.593712 5840 raft_consensus.cc:374] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.593801 5840 raft_consensus.cc:729] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.594056 5840 consensus_queue.cc:260] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.594977 5843 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.595320 5843 ts_tablet_manager.cc:1367] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.101s user 0.006s sys 0.008s
I20240507 08:12:15.596371 5843 raft_consensus.cc:348] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.596632 5843 raft_consensus.cc:374] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.596730 5843 raft_consensus.cc:729] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.596962 5840 ts_tablet_manager.cc:1398] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20240507 08:12:15.596997 5843 consensus_queue.cc:260] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.597344 5840 tablet_bootstrap.cc:492] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:15.597635 5843 ts_tablet_manager.cc:1398] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:15.598954 6675 raft_consensus.cc:2781] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.599036 6513 raft_consensus.cc:2258] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.599103 6675 raft_consensus.cc:482] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.599177 6675 raft_consensus.cc:3037] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.599572 6513 ts_tablet_manager.cc:1875] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.601802 6675 raft_consensus.cc:504] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.602502 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe7a7c1bee444c75a4ed6215db0e25bd consensus state has no leader
I20240507 08:12:15.602646 6675 leader_election.cc:288] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.603511 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_54152e52f0df4b3281c833102a7d9aff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.604138 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54152e52f0df4b3281c833102a7d9aff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.605996 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_012b7342d513481a9f0bde95ef70b498" } new_table_name: "default.Table_939ac1a3eef54238b67882f7284a2a94" modify_external_catalogs: true
I20240507 08:12:15.607239 5839 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:15.607856 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9862857aff3b4392b5c75945a0969cf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.608328 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9862857aff3b4392b5c75945a0969cf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.609694 5437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.609916 5437 raft_consensus.cc:3037] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.612408 5437 raft_consensus.cc:2454] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:15.612741 5656 leader_election.cc:302] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.612988 6675 raft_consensus.cc:2781] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.613099 6675 raft_consensus.cc:686] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.613332 6675 consensus_queue.cc:237] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.617326 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 66699.2us
I20240507 08:12:15.617918 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b6e68364238c4b66b8c0de0f4a2ea855 (table default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]) successfully deleted
W20240507 08:12:15.618532 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 070fdbf0b00b4582a184729c2f2a4b69 because the tablet was not found. No further retry: Not found: Tablet not found: 070fdbf0b00b4582a184729c2f2a4b69
W20240507 08:12:15.618851 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5c49745fce2949d1909baf7583b157a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5c49745fce2949d1909baf7583b157a7 already in progress: creating tablet
W20240507 08:12:15.619050 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0c51ec8f693e4f108100c75470ead2a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0c51ec8f693e4f108100c75470ead2a9 already in progress: creating tablet
W20240507 08:12:15.619242 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7fbe0f58bad541e7bcfb611105328d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fbe0f58bad541e7bcfb611105328d56 already in progress: creating tablet
I20240507 08:12:15.620491 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.622822 5377 leader_election.cc:302] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.623418 7469 raft_consensus.cc:2726] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:15.626902 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4ab977deaf2b47b281a5d9ae655f0f43" } modify_external_catalogs: true
I20240507 08:12:15.627002 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.627110 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4ab977deaf2b47b281a5d9ae655f0f43" } modify_external_catalogs: true
I20240507 08:12:15.628288 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e5439844f384513b4719cfe4f4e5d9b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.629062 7418 raft_consensus.cc:482] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.629205 7418 raft_consensus.cc:504] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.629949 7418 leader_election.cc:288] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.630134 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_43d29dc135d54b5bb43129f3ee487d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.630839 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43d29dc135d54b5bb43129f3ee487d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:15.630867 5520 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 150efae7834740b798af86ae1dcfe445
W20240507 08:12:15.631862 5517 leader_election.cc:341] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 150efae7834740b798af86ae1dcfe445
W20240507 08:12:15.631974 5520 consensus_peers.cc:485] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79524df2415146ab926425c42068626c. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.631991 5517 leader_election.cc:302] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:15.633044 7418 raft_consensus.cc:2726] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:15.639582 7079 ts_tablet_manager.cc:1888] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.639707 7079 log.cc:1196] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/44ec0526e1914221bc22b3e75ea01e0d
I20240507 08:12:15.640038 7079 ts_tablet_manager.cc:1909] T 44ec0526e1914221bc22b3e75ea01e0d P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:15.640434 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1040) took 2644 ms. Trace:
I20240507 08:12:15.640508 7079 rpcz_store.cc:276] 0507 08:12:12.996440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.996489 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:15.349884 (+2353395us) ts_tablet_manager.cc:1120] Deleting tablet 44ec0526e1914221bc22b3e75ea01e0d
0507 08:12:15.349898 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.453932 (+104034us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.639473 (+185541us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.640414 (+ 941us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2353192}
I20240507 08:12:15.641038 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 44ec0526e1914221bc22b3e75ea01e0d (table default.Table_ede2eb6eb3b247c2ac451571f36e8e29 [id=e222b90053c54c73bc26953648fc9f20]) successfully deleted
I20240507 08:12:15.641366 7079 tablet_replica.cc:330] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:15.641634 7079 raft_consensus.cc:2229] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.641880 7079 raft_consensus.cc:2258] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.642465 7079 ts_tablet_manager.cc:1875] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.642554 6513 ts_tablet_manager.cc:1888] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240507 08:12:15.650070 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d432a59bdd344ddaaf710de73b543e6 consensus state has no leader
I20240507 08:12:15.650430 5843 tablet_bootstrap.cc:492] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.648687 6513 log.cc:1196] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/28e5a380f2784e3a806afcbb5cd2519a
W20240507 08:12:15.651468 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.063s user 0.003s sys 0.000s
I20240507 08:12:15.653187 6513 ts_tablet_manager.cc:1909] T 28e5a380f2784e3a806afcbb5cd2519a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:15.653357 5843 tablet_bootstrap.cc:654] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.654712 5841 ts_tablet_manager.cc:1888] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.654816 5841 log.cc:1196] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:12:15.655138 5841 ts_tablet_manager.cc:1909] T 831153d2b89e43f8b08b086f7a8f3a54 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.657001 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b5ae6b3c712d4df3a94846fb9a4ef837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.657641 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5ae6b3c712d4df3a94846fb9a4ef837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.658020 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1008) took 4423 ms. Trace:
I20240507 08:12:15.658841 6513 rpcz_store.cc:276] 0507 08:12:11.234446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.234505 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:15.561563 (+4327058us) ts_tablet_manager.cc:1120] Deleting tablet 28e5a380f2784e3a806afcbb5cd2519a
0507 08:12:15.561591 (+ 28us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.616306 (+ 54715us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.642414 (+ 26108us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.658005 (+ 15591us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":36891,"tablet-delete.queue_time_us":4326855}
I20240507 08:12:15.656335 5840 tablet_bootstrap.cc:654] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:15.657500 5654 leader_election.cc:341] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:15.661561 5654 leader_election.cc:341] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.662282 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 3 got 0
I20240507 08:12:15.663322 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 28e5a380f2784e3a806afcbb5cd2519a (table default.Table_8a58f4e04c4147368d43d162c9613c9f [id=0e33fa8f8fff4e09916ca2783a4903e7]) successfully deleted
I20240507 08:12:15.665083 5840 tablet_bootstrap.cc:492] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:15.665273 5840 ts_tablet_manager.cc:1367] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.068s user 0.004s sys 0.000s
I20240507 08:12:15.666108 5840 raft_consensus.cc:348] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.666375 5840 raft_consensus.cc:374] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.666481 5840 raft_consensus.cc:729] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.666718 5840 consensus_queue.cc:260] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.666975 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet be1b36ca5b6041d780e851f137d3a4c9 (table default.Table_a3ed2876b6214485a0f2bbfb0d196d5a [id=6325b34aec0b45d981c7a0825903409a], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet be1b36ca5b6041d780e851f137d3a4c9 consensus state has no leader
I20240507 08:12:15.667269 5840 ts_tablet_manager.cc:1398] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:15.667551 5840 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:15.667538 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 62895377a8ac4506a2f4e10039dbdd74
I20240507 08:12:15.668501 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.668709 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.663048 5259 catalog_manager.cc:5484] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:15.669278 5840 tablet_bootstrap.cc:654] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.669219 5259 catalog_manager.cc:5484] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:15.669678 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:15.669957 5259 catalog_manager.cc:5484] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:15.670385 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:15.671188 5259 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_a63ec02af6ab4e85830582d44267f8c1 [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 1 got 0
I20240507 08:12:15.672094 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 975f22194aff40609a3cfa63b9b3aca2
I20240507 08:12:15.673478 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d] alter complete (version 2)
I20240507 08:12:15.676043 5839 tablet_bootstrap.cc:654] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.675069 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.678136 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.659968 6513 tablet_replica.cc:330] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:15.679962 6513 raft_consensus.cc:2229] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:15.680344 6513 raft_consensus.cc:2258] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.662979 7469 consensus_queue.cc:1035] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.668932 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 62895377a8ac4506a2f4e10039dbdd74 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.682675 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1027) took 3965 ms. Trace:
I20240507 08:12:15.673031 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.682746 5841 rpcz_store.cc:276] 0507 08:12:11.717488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.718299 (+ 811us) service_pool.cc:225] Handling call
0507 08:12:15.556134 (+3837835us) ts_tablet_manager.cc:1120] Deleting tablet 831153d2b89e43f8b08b086f7a8f3a54
0507 08:12:15.556144 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.588807 (+ 32663us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.654597 (+ 65790us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.682656 (+ 28059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3837663}
I20240507 08:12:15.685343 6513 ts_tablet_manager.cc:1875] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.688906 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.689254 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 150efae7834740b798af86ae1dcfe445 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.690138 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d617213b959452ab6519809e56db128 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.690151 5564 tablet_service.cc:1514] Processing DeleteTablet for tablet b0b9f85cd78748d3afce83b64b3f823b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.690469 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 1037c3c0af5e464bb23a311e552fcf21 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.677664 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.678624 5425 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.695623 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet e298f11ff9f54e48b8de435f6849f39a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.678746 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.678853 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.678951 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.679031 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.679124 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.673633 5843 tablet_bootstrap.cc:492] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.696084 5843 ts_tablet_manager.cc:1367] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.001s
I20240507 08:12:15.696440 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 831153d2b89e43f8b08b086f7a8f3a54 (table default.Table_1c03c02bf7c44286b5e1273ff2b738c0 [id=4cd92a7dd38946788551d84e24279035]) successfully deleted
I20240507 08:12:15.696955 5843 raft_consensus.cc:348] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.697155 5843 raft_consensus.cc:374] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.697227 5843 raft_consensus.cc:729] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.697465 5843 consensus_queue.cc:260] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.698163 5843 ts_tablet_manager.cc:1398] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:15.698607 5843 tablet_bootstrap.cc:492] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.700341 5843 tablet_bootstrap.cc:654] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.702446 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_530de072d6794063a4670b0748d8f4f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.702916 7469 raft_consensus.cc:482] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.703028 7469 raft_consensus.cc:504] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:15.703111 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_530de072d6794063a4670b0748d8f4f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.703699 7469 leader_election.cc:288] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.704586 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:15.705168 5378 leader_election.cc:341] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.709842 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e3dfd4eec9cf49a3a875332a653befd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.710315 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3dfd4eec9cf49a3a875332a653befd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.711645 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5cc40acfd8e04ef1813f07ec723496d6" } modify_external_catalogs: true
I20240507 08:12:15.711786 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5cc40acfd8e04ef1813f07ec723496d6" } modify_external_catalogs: true
I20240507 08:12:15.712972 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet b6e68364238c4b66b8c0de0f4a2ea855 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:15.713855 5841 tablet_replica.cc:330] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.714078 5841 raft_consensus.cc:2229] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.714278 5841 raft_consensus.cc:2258] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.715853 5843 tablet_bootstrap.cc:492] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.716101 5843 ts_tablet_manager.cc:1367] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240507 08:12:15.717372 5843 raft_consensus.cc:348] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.717625 5843 raft_consensus.cc:374] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.718225 5843 raft_consensus.cc:729] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.718549 5843 consensus_queue.cc:260] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.719202 5843 ts_tablet_manager.cc:1398] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:12:15.719511 5843 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.720388 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ddcb720c09c5486995391837fc54e7ce
I20240507 08:12:15.721040 5843 tablet_bootstrap.cc:654] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.705005 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.713299 7394 raft_consensus.cc:482] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.721985 7394 raft_consensus.cc:504] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.722990 7394 leader_election.cc:288] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.723315 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb720c09c5486995391837fc54e7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.726508 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f9b60e7fd01c4c96911fff6bf62ade96
I20240507 08:12:15.727161 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.728543 7079 ts_tablet_manager.cc:1888] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:15.728649 7079 log.cc:1196] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/7ba2e8d1e01047fd9cc1c192e9628750
I20240507 08:12:15.728953 7079 ts_tablet_manager.cc:1909] T 7ba2e8d1e01047fd9cc1c192e9628750 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:15.729319 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1041) took 2533 ms. Trace:
I20240507 08:12:15.729403 7079 rpcz_store.cc:276] 0507 08:12:13.195896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.196592 (+ 696us) service_pool.cc:225] Handling call
0507 08:12:15.641316 (+2444724us) ts_tablet_manager.cc:1120] Deleting tablet 7ba2e8d1e01047fd9cc1c192e9628750
0507 08:12:15.641328 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.725679 (+ 84351us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.728466 (+ 2787us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.729302 (+ 836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2444505}
I20240507 08:12:15.730888 7394 consensus_queue.cc:1035] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.730512 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1042) took 2396 ms. Trace:
I20240507 08:12:15.731720 7079 rpcz_store.cc:276] 0507 08:12:13.333935 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.333989 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:15.730401 (+2396412us) ts_tablet_manager.cc:1120] Deleting tablet 34adee00b529424b9f274c61ce73b0bc
0507 08:12:15.730411 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.730498 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2396238}
I20240507 08:12:15.732081 7079 tablet_replica.cc:330] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:15.732299 7079 raft_consensus.cc:2229] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.732514 7079 raft_consensus.cc:2258] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.732535 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.732860 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 156673b3031d436db6391ff971641b84 (DEFAULT_TABLE table=default.Table_2dc142f4782144e49dfe1eb829d61be8 [id=afbc0213cea84332bdf16fa56af568f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.733253 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156673b3031d436db6391ff971641b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.733506 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb720c09c5486995391837fc54e7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.741130 7079 ts_tablet_manager.cc:1875] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.742949 5840 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:15.743109 5840 ts_tablet_manager.cc:1367] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20240507 08:12:15.743872 5840 raft_consensus.cc:348] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.744065 5840 raft_consensus.cc:374] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.723610 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14487e625374060b118c9cce4500b3f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:15.744254 5445 raft_consensus.cc:2454] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:15.744760 5656 leader_election.cc:302] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.730154 5841 ts_tablet_manager.cc:1875] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.745143 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 8868c2ff8b434200b427d4198b0374d9 (DEFAULT_TABLE table=default.Table_7c08a4452a324ca5b3cc664cfef163ad [id=a345eda955bd4c168843fae6c0eb9943]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.731411 5377 leader_election.cc:341] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.745326 5377 leader_election.cc:302] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:15.745314 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec8ef4c7f74420088cdb05294764b99 (DEFAULT_TABLE table=default.Table_47367ed796754576afd6b024153d847f [id=2fa2e4cc886c443fa9bb6cbe525e523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.745501 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868c2ff8b434200b427d4198b0374d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.745708 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec8ef4c7f74420088cdb05294764b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.745870 7408 raft_consensus.cc:2726] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:15.745998 5695 tablet_service.cc:1467] Processing CreateTablet for tablet a54aa68a2b364d5f8f4f4c217bb3e167 (DEFAULT_TABLE table=default.Table_0a81eec1a1d34fa8bdbe9fac1d360516 [id=8ae9a7147ada4165b5d6197138892556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.746280 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54aa68a2b364d5f8f4f4c217bb3e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.747285 7408 raft_consensus.cc:482] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.747403 7408 raft_consensus.cc:504] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.747960 7408 leader_election.cc:288] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.732482 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet ddcb720c09c5486995391837fc54e7ce with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.749660 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.732940 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b60e7fd01c4c96911fff6bf62ade96 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.749665 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.733441 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 156673b3031d436db6391ff971641b84 (DEFAULT_TABLE table=default.Table_2dc142f4782144e49dfe1eb829d61be8 [id=afbc0213cea84332bdf16fa56af568f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.750237 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.750324 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156673b3031d436db6391ff971641b84. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.750665 5377 leader_election.cc:341] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.752241 7394 raft_consensus.cc:2781] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:15.752370 7394 raft_consensus.cc:482] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:15.752492 7394 raft_consensus.cc:3037] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.753770 5698 tablet_service.cc:1467] Processing CreateTablet for tablet eb1213d587ea47bfb55a6cfa413430e8 (DEFAULT_TABLE table=default.Table_44bcd391ca2f4d08a04fd38c02b6f957 [id=a1086570b84e4bfe9c28b02a0e4ca9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.754149 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1213d587ea47bfb55a6cfa413430e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.753875 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb5555330ea49a6b560e90d54ee5f5f (DEFAULT_TABLE table=default.Table_43a808c00e4b4c99b2a63595d71acbd4 [id=a88c66785dd04de5ac0150bf0aa7ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.758347 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb5555330ea49a6b560e90d54ee5f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.759305 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 38fc0180e9d34d11885b819529407930
I20240507 08:12:15.762141 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 38fc0180e9d34d11885b819529407930 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:15.764362 5656 consensus_peers.cc:485] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cd4379b6fded4a52843f55f27ae67121. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.764364 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb5555330ea49a6b560e90d54ee5f5f (DEFAULT_TABLE table=default.Table_43a808c00e4b4c99b2a63595d71acbd4 [id=a88c66785dd04de5ac0150bf0aa7ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.764729 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb5555330ea49a6b560e90d54ee5f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.766291 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.766482 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet b6e68364238c4b66b8c0de0f4a2ea855 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a015bf84a454d49b6497c16ed15c84a) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.766671 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.766772 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.766979 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.766948 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 33a58839996b414d805c6b2873254562 (DEFAULT_TABLE table=default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0 [id=c3ee217cbe574081a5ad8987ad9fc370]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.767202 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.767321 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a58839996b414d805c6b2873254562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.767371 5841 ts_tablet_manager.cc:1888] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.767459 5841 log.cc:1196] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:15.768368 5685 tablet_service.cc:1467] Processing CreateTablet for tablet c33ba079a17b4eafb8031be383c91106 (DEFAULT_TABLE table=default.Table_5c436dbf3c0f4f879e1d3a57d6405594 [id=c6e925419297458087d106a24ce71803]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.768679 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ba079a17b4eafb8031be383c91106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.771270 5841 ts_tablet_manager.cc:1909] T 85dc58133d2b4f5496b4e570ae6494c0 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.772301 5691 tablet_service.cc:1467] Processing CreateTablet for tablet a1a06e10c76546b190adbc63820c080e (DEFAULT_TABLE table=default.Table_e7a8a764bfdf4f358f30d679d0bbf5d6 [id=dd79de02fb044d2cab9c8f11e64097b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.772375 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1029) took 3920 ms. Trace:
I20240507 08:12:15.772445 5841 rpcz_store.cc:276] 0507 08:12:11.851607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.860029 (+ 8422us) service_pool.cc:225] Handling call
0507 08:12:15.713803 (+3853774us) ts_tablet_manager.cc:1120] Deleting tablet 85dc58133d2b4f5496b4e570ae6494c0
0507 08:12:15.713814 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.756093 (+ 42279us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.767287 (+ 11194us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.772356 (+ 5069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3853483}
I20240507 08:12:15.772615 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a06e10c76546b190adbc63820c080e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.772857 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1031) took 3898 ms. Trace:
I20240507 08:12:15.774778 5841 rpcz_store.cc:276] 0507 08:12:11.874206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.874256 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:15.772730 (+3898474us) ts_tablet_manager.cc:1120] Deleting tablet e3783fc7fe434696898ba14ab77a9058
0507 08:12:15.772739 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.772841 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3898281}
I20240507 08:12:15.775135 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1030) took 3901 ms. Trace:
I20240507 08:12:15.775203 5841 rpcz_store.cc:276] 0507 08:12:11.873246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.877632 (+ 4386us) service_pool.cc:225] Handling call
0507 08:12:15.775015 (+3897383us) ts_tablet_manager.cc:1120] Deleting tablet aafd1dc2eaf84390b072f01a9e76232d
0507 08:12:15.775024 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.775119 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3893000}
I20240507 08:12:15.776319 5692 tablet_service.cc:1467] Processing CreateTablet for tablet c59886326a48434fb737976d9f571cd3 (DEFAULT_TABLE table=default.Table_2cd819bfa63841308bf0554cd623e25a [id=c6905d870b9b45308aaca4db0d9f34ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.776666 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59886326a48434fb737976d9f571cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.777067 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 818246c9039e4e00aed16701ab9b8271 (DEFAULT_TABLE table=default.Table_eb491699584f4d18ad9cb26ed87fd99b [id=9f57749f2e8e4390af59816f6ab59067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.777410 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818246c9039e4e00aed16701ab9b8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.777426 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1033) took 3886 ms. Trace:
I20240507 08:12:15.777494 5841 rpcz_store.cc:276] 0507 08:12:11.890496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.890551 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:15.775393 (+3884842us) ts_tablet_manager.cc:1120] Deleting tablet 323b96ae7d1e471998a39f3be0857ce7
0507 08:12:15.775402 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.777410 (+ 2008us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3884273}
I20240507 08:12:15.777805 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1032) took 3887 ms. Trace:
I20240507 08:12:15.777865 5841 rpcz_store.cc:276] 0507 08:12:11.890339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.895488 (+ 5149us) service_pool.cc:225] Handling call
0507 08:12:15.777708 (+3882220us) ts_tablet_manager.cc:1120] Deleting tablet cacfe980b2aa4a60ad05bbe102a94266
0507 08:12:15.777715 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.777793 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3879233}
I20240507 08:12:15.778105 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1034) took 3877 ms. Trace:
I20240507 08:12:15.779418 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 8b47c57b0f964c7894461a594bb22539 (DEFAULT_TABLE table=default.Table_1f3f2ed28af24116b1d81eecee74897c [id=aa491803c5144419aa6df4c2f0a7c8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.779700 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b47c57b0f964c7894461a594bb22539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.780543 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf9b2427a24eaf9d9ab14a00bed5df (DEFAULT_TABLE table=default.Table_ba61f9a9c956447589c8e0db1eaa6f6f [id=5d80f7b5b3e7459692cbc3140f7822fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.780820 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf9b2427a24eaf9d9ab14a00bed5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.781904 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee977e6c7034aaa8148ec6bcfeedae2 (DEFAULT_TABLE table=default.Table_40e14999737746f0984e6386706a3d9c [id=2bbd1eed5ca74f24bbb08b803e9816e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.782264 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee977e6c7034aaa8148ec6bcfeedae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.778168 5841 rpcz_store.cc:276] 0507 08:12:11.900922 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.901161 (+ 239us) service_pool.cc:225] Handling call
0507 08:12:15.778030 (+3876869us) ts_tablet_manager.cc:1120] Deleting tablet 6ad4ed7878454fea930b3982bcb6eed4
0507 08:12:15.778037 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.778094 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3876649}
I20240507 08:12:15.786027 5841 tablet_replica.cc:330] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.785988 7394 raft_consensus.cc:504] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.786280 5841 raft_consensus.cc:2229] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.786451 5841 raft_consensus.cc:2258] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.787420 5690 tablet_service.cc:1467] Processing CreateTablet for tablet fce52cf5871c402caeb769d1ca4c408a (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.788479 5841 ts_tablet_manager.cc:1875] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.790242 7394 leader_election.cc:288] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.790302 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14487e625374060b118c9cce4500b3f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:15.790486 5445 raft_consensus.cc:3037] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:15.792894 5445 raft_consensus.cc:2454] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:15.793339 5656 leader_election.cc:302] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:15.793269 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c63d7d02e1734694abdd64ac83426682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.793495 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1e121ea477624cf8a2fea778d5ca71ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.793905 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c63d7d02e1734694abdd64ac83426682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:15.793964 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e121ea477624cf8a2fea778d5ca71ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.795076 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7fa1f6e175bf4f0d9146ed3ca065146a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.795262 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 140b4601f1a043329c4de612a3ae244c (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.795480 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fa1f6e175bf4f0d9146ed3ca065146a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.795670 5689 tablet_service.cc:1467] Processing CreateTablet for tablet dd97542e3db0413c8272ff2971801be7 (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.796427 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a99b17a89c46447ea9e6544b2f7eb4d4" } new_table_name: "default.Table_73b2cacc437f4f8791c29d803ea49436" modify_external_catalogs: true
I20240507 08:12:15.797948 7394 raft_consensus.cc:2781] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:15.798066 7394 raft_consensus.cc:686] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:15.798287 7394 consensus_queue.cc:237] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.799314 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.799893 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5e6b208d88324e1d913e9c07d26d71e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:15.800066 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_605d0c8dfbc846a69225b3abf59b892e" } new_table_name: "default.Table_d09239195ced4bc89bc7103f1a4bc28d" modify_external_catalogs: true
I20240507 08:12:15.800160 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 8868c2ff8b434200b427d4198b0374d9 (DEFAULT_TABLE table=default.Table_7c08a4452a324ca5b3cc664cfef163ad [id=a345eda955bd4c168843fae6c0eb9943]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.800325 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e6b208d88324e1d913e9c07d26d71e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.800522 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868c2ff8b434200b427d4198b0374d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.801332 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_73e379813c044602aa406c9ac47557ce" } new_table_name: "default.Table_12d7cdd24a7b4055ae8572cb30f25040" modify_external_catalogs: true
I20240507 08:12:15.801802 5421 tablet_service.cc:1467] Processing CreateTablet for tablet a54aa68a2b364d5f8f4f4c217bb3e167 (DEFAULT_TABLE table=default.Table_0a81eec1a1d34fa8bdbe9fac1d360516 [id=8ae9a7147ada4165b5d6197138892556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.802137 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54aa68a2b364d5f8f4f4c217bb3e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.808211 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec8ef4c7f74420088cdb05294764b99 (DEFAULT_TABLE table=default.Table_47367ed796754576afd6b024153d847f [id=2fa2e4cc886c443fa9bb6cbe525e523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.808688 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec8ef4c7f74420088cdb05294764b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.809058 5843 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:15.809054 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f89efb8414dc49369d361adf908927e6 consensus state has no leader
I20240507 08:12:15.809260 5843 ts_tablet_manager.cc:1367] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.002s
I20240507 08:12:15.809965 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_996bd171383346f6b81bdcfc316f9ee4" } new_table_name: "default.Table_4638b813c41e48b4b3d7afef76128b8a" modify_external_catalogs: true
I20240507 08:12:15.810312 5843 raft_consensus.cc:348] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.811491 5843 raft_consensus.cc:374] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.811615 5843 raft_consensus.cc:729] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.811985 7469 raft_consensus.cc:482] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.811878 5843 consensus_queue.cc:260] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.812103 7469 raft_consensus.cc:504] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.812511 5843 ts_tablet_manager.cc:1398] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240507 08:12:15.812685 7469 leader_election.cc:288] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.812742 5843 tablet_bootstrap.cc:492] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:15.814790 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e05cf76c9934d4881c3296fd6f8328c consensus state has no leader
I20240507 08:12:15.814803 7333 consensus_queue.cc:1035] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:15.815100 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dba4967c141f4078b6f24d4b71165906 consensus state has no leader
I20240507 08:12:15.819505 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 38fc0180e9d34d11885b819529407930 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:15.819707 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb968d91481464c830133873b16b8e0 (DEFAULT_TABLE table=default.Table_42a34eaa8a514f349f1a092abfff155f [id=abbc1d232ee54624a3cd2e6ca6e2433a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.819844 5843 tablet_bootstrap.cc:654] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.820065 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb968d91481464c830133873b16b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.821499 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5f9846e0eab74388af231e9942a6881a consensus state has no leader
I20240507 08:12:15.821816 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.822351 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce52cf5871c402caeb769d1ca4c408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.823091 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140b4601f1a043329c4de612a3ae244c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:15.823155 5377 leader_election.cc:341] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.825513 5427 tablet_service.cc:1467] Processing CreateTablet for tablet eb1213d587ea47bfb55a6cfa413430e8 (DEFAULT_TABLE table=default.Table_44bcd391ca2f4d08a04fd38c02b6f957 [id=a1086570b84e4bfe9c28b02a0e4ca9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.825850 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1213d587ea47bfb55a6cfa413430e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.827746 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97542e3db0413c8272ff2971801be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.830075 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 7c73a5096ae149d6ba2b792f1d6966d5 (DEFAULT_TABLE table=default.Table_2f032a109d8d4902bd1611542a32e9d9 [id=c8bc7ac4fed1481ca19cee47f8310763]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.830472 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c73a5096ae149d6ba2b792f1d6966d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.832019 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 919078225a45448a9b0013e340c041b9 (DEFAULT_TABLE table=default.Table_2915a6cc4171446abe445f6c4fb1eb44 [id=3c6228d5a39a41e78cbff2a8ee8e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.832454 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919078225a45448a9b0013e340c041b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.832587 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c35d3c43d02b421186d2a6a8548ac31f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.833102 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c35d3c43d02b421186d2a6a8548ac31f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.838171 5699 tablet_service.cc:1467] Processing CreateTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 (DEFAULT_TABLE table=default.Table_091c1a53ece34eb3bbc8265fa45f9088 [id=e48fba499b3f47728a1a2fc848800e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.840128 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42f6e958dd94e3cbecd5c4bd6a6d829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.844071 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b1919d9d46d884c5784accbb5304 (DEFAULT_TABLE table=default.Table_087f9c936f35417f854ffeeb097ca835 [id=3b47c5283e8f42a59dfea9fea27d08e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.844408 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b1919d9d46d884c5784accbb5304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.846900 5691 tablet_service.cc:1467] Processing CreateTablet for tablet b967f56b73aa4c4eb838c42c831b9712 (DEFAULT_TABLE table=default.Table_f2867d292c9d45a3b77c426279d29c32 [id=361b5acb391340c6a150f3f081cfedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.847127 5683 tablet_service.cc:1467] Processing CreateTablet for tablet d4216050db0146e59aac37b6e8b0018c (DEFAULT_TABLE table=default.Table_7a8bd4e6c9d149189ae4d0f6877c8cdd [id=2c7582e1958e4126b24894a136730344]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.847363 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967f56b73aa4c4eb838c42c831b9712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.847604 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4216050db0146e59aac37b6e8b0018c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.850003 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 67d479e3dd17488e914adf01e630169e (DEFAULT_TABLE table=default.Table_7d3cd26b01594f059ae92b67fe87d846 [id=74cae62c166e44339803e2700dc5da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.850298 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d479e3dd17488e914adf01e630169e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.850631 5841 ts_tablet_manager.cc:1888] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.850728 5841 log.cc:1196] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/33875331045146a9a02559a0b37d5c6f
I20240507 08:12:15.851038 5841 ts_tablet_manager.cc:1909] T 33875331045146a9a02559a0b37d5c6f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.852357 5843 tablet_bootstrap.cc:492] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.853798 5697 tablet_service.cc:1467] Processing CreateTablet for tablet f921a7e89a874c64817bff535ad4eda7 (DEFAULT_TABLE table=default.Table_97f7aa309ac14b62b69cbe42ce8ff48b [id=947babe79251496d855060b66eecef30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.854068 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f921a7e89a874c64817bff535ad4eda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.855747 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1035) took 3925 ms. Trace:
I20240507 08:12:15.855823 5841 rpcz_store.cc:276] 0507 08:12:11.929960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.930347 (+ 387us) service_pool.cc:225] Handling call
0507 08:12:15.785985 (+3855638us) ts_tablet_manager.cc:1120] Deleting tablet 33875331045146a9a02559a0b37d5c6f
0507 08:12:15.785994 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.830596 (+ 44602us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.850539 (+ 19943us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.855729 (+ 5190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3855462}
I20240507 08:12:15.856896 5841 tablet_replica.cc:330] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.857067 5841 raft_consensus.cc:2229] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:15.857205 5841 raft_consensus.cc:2258] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.857719 5841 ts_tablet_manager.cc:1875] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.857692 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 994f50ea19a2445288663b7d7b7ad291 (DEFAULT_TABLE table=default.Table_bc1f5c0697f04fbba453282facb1a6a1 [id=41cc0579a0124faa915d338c27ba480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.857973 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994f50ea19a2445288663b7d7b7ad291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.860142 5683 tablet_service.cc:1467] Processing CreateTablet for tablet f1980322500f42b99180b86cc259ca3f (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.860433 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980322500f42b99180b86cc259ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.860662 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_510d804e03f54c96b5321a44a0b1dd38" } new_table_name: "default.Table_ca95e0ec1d914306aa594547a64296b0" modify_external_catalogs: true
I20240507 08:12:15.861532 5687 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ece8042754f4e9877f286f02ffd64 (DEFAULT_TABLE table=default.Table_ca9f79f6fd724517a82be3b0bf66dcf8 [id=e91aa5cee994450d8a06443bfc049be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.861866 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ece8042754f4e9877f286f02ffd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.862373 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4ea29887ff37462fa4e6f2ca98542433"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.862851 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ea29887ff37462fa4e6f2ca98542433 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.866525 5843 ts_tablet_manager.cc:1367] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.000s
I20240507 08:12:15.867108 7469 raft_consensus.cc:482] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.869957 7469 raft_consensus.cc:504] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.870616 7469 leader_election.cc:288] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:15.870761 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_848f877aafbe4022bb2a1ec887d8245b" } modify_external_catalogs: true
I20240507 08:12:15.870947 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_848f877aafbe4022bb2a1ec887d8245b" } modify_external_catalogs: true
I20240507 08:12:15.871771 5843 raft_consensus.cc:348] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.872421 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0d2215c8f3fd4c46ab9709243c307fac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.872895 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d2215c8f3fd4c46ab9709243c307fac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.873796 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:15.873872 5843 raft_consensus.cc:374] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.873941 5843 raft_consensus.cc:729] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:15.874135 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 90dcd7eee62b458ea4661a05a1bfcac4 consensus state has no leader
I20240507 08:12:15.874161 5843 consensus_queue.cc:260] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.875123 5843 ts_tablet_manager.cc:1398] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240507 08:12:15.876019 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e22a29266ff54905b0152cf90c36bff0" } modify_external_catalogs: true
I20240507 08:12:15.876171 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e22a29266ff54905b0152cf90c36bff0" } modify_external_catalogs: true
I20240507 08:12:15.876483 5843 tablet_bootstrap.cc:492] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.877501 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9ff8a98ef41745b598c1d26d9c862529"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.877961 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ff8a98ef41745b598c1d26d9c862529 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:15.881302 5377 leader_election.cc:341] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:15.883005 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 0ddb7dd28f43488581d3a75bf1d68a8a
I20240507 08:12:15.883937 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b15c8ad2a8324c75a1ef49e219b962cf
I20240507 08:12:15.883915 7101 consensus_queue.cc:1035] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:15.885447 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.885761 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.886247 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 85dc58133d2b4f5496b4e570ae6494c0 (table default.Table_f7a84b84863d462192343bcb92bd888d [id=5714d419aa014285ac6779ef49cc29fa]) successfully deleted
W20240507 08:12:15.893936 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e3783fc7fe434696898ba14ab77a9058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3783fc7fe434696898ba14ab77a9058 already in progress: creating tablet
W20240507 08:12:15.894824 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet aafd1dc2eaf84390b072f01a9e76232d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafd1dc2eaf84390b072f01a9e76232d already in progress: creating tablet
W20240507 08:12:15.899405 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 323b96ae7d1e471998a39f3be0857ce7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323b96ae7d1e471998a39f3be0857ce7 already in progress: creating tablet
I20240507 08:12:15.899605 5843 tablet_bootstrap.cc:654] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:15.905213 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cacfe980b2aa4a60ad05bbe102a94266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cacfe980b2aa4a60ad05bbe102a94266 already in progress: creating tablet
W20240507 08:12:15.905401 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6ad4ed7878454fea930b3982bcb6eed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ad4ed7878454fea930b3982bcb6eed4 already in progress: creating tablet
I20240507 08:12:15.911054 5843 tablet_bootstrap.cc:492] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.912156 5843 ts_tablet_manager.cc:1367] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20240507 08:12:15.914038 5843 raft_consensus.cc:348] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.914306 5843 raft_consensus.cc:374] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.914391 5843 raft_consensus.cc:729] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.914614 5843 consensus_queue.cc:260] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:15.915421 5843 ts_tablet_manager.cc:1398] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:15.916313 5841 ts_tablet_manager.cc:1888] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.922608 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_03e9834c587b47ff9dcd33041dba5abf" } modify_external_catalogs: true
I20240507 08:12:15.922816 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_03e9834c587b47ff9dcd33041dba5abf" } modify_external_catalogs: true
I20240507 08:12:15.898746 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb968d91481464c830133873b16b8e0 (DEFAULT_TABLE table=default.Table_42a34eaa8a514f349f1a092abfff155f [id=abbc1d232ee54624a3cd2e6ca6e2433a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.929359 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb968d91481464c830133873b16b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.930423 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 7ba2e8d1e01047fd9cc1c192e9628750 (table default.Table_342a9c864d774adf993c547b7e2f79d1 [id=8e24de9c1ceb43308ab34b72f08bf18c]) successfully deleted
W20240507 08:12:15.930689 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 34adee00b529424b9f274c61ce73b0bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34adee00b529424b9f274c61ce73b0bc already in progress: creating tablet
I20240507 08:12:15.932116 5256 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 3 got 0
I20240507 08:12:15.932945 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c20c60fe0aad4cb3b12d04ad1a87d9f4
I20240507 08:12:15.934160 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.934367 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.934511 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:15.934530 5839 tablet_bootstrap.cc:492] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:15.934652 5839 ts_tablet_manager.cc:1367] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.354s user 0.004s sys 0.000s
I20240507 08:12:15.934542 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_05652d1f4f574b078ce9726ea093c0eb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.935079 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05652d1f4f574b078ce9726ea093c0eb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.935286 5839 raft_consensus.cc:348] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.935460 5839 raft_consensus.cc:374] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.935500 5839 raft_consensus.cc:729] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.935672 5839 consensus_queue.cc:260] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.936100 5839 ts_tablet_manager.cc:1398] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240507 08:12:15.936373 5839 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:15.937800 7480 tablet.cc:1722] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:15.937875 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7ef0ec9add454eb28c8bb9725ee5274f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:15.938313 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ef0ec9add454eb28c8bb9725ee5274f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:15.939312 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9ff8a98ef41745b598c1d26d9c862529" } modify_external_catalogs: true
I20240507 08:12:15.939463 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9ff8a98ef41745b598c1d26d9c862529" } modify_external_catalogs: true
I20240507 08:12:15.942341 5839 tablet_bootstrap.cc:654] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.948962 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 156673b3031d436db6391ff971641b84 (DEFAULT_TABLE table=default.Table_2dc142f4782144e49dfe1eb829d61be8 [id=afbc0213cea84332bdf16fa56af568f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.952308 5841 log.cc:1196] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/dc74342c8aee4017a212aa88a7f3dc06
I20240507 08:12:15.953274 5841 ts_tablet_manager.cc:1909] T dc74342c8aee4017a212aa88a7f3dc06 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:15.953670 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1036) took 3970 ms. Trace:
I20240507 08:12:15.953809 5841 rpcz_store.cc:276] 0507 08:12:11.983118 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.983171 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:15.856852 (+3873681us) ts_tablet_manager.cc:1120] Deleting tablet dc74342c8aee4017a212aa88a7f3dc06
0507 08:12:15.856861 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.911044 (+ 54183us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.916203 (+ 5159us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.953653 (+ 37450us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3873480}
I20240507 08:12:15.954381 5841 tablet_replica.cc:330] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:15.954604 5841 raft_consensus.cc:2229] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:15.955104 5841 raft_consensus.cc:2258] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:15.956223 7418 raft_consensus.cc:482] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:15.932468 7481 tablet.cc:1722] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:15.938417 5843 tablet_bootstrap.cc:492] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.962514 5843 tablet_bootstrap.cc:654] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.966073 5843 tablet_bootstrap.cc:492] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:15.965975 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14487e625374060b118c9cce4500b3f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.965999 7418 raft_consensus.cc:504] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.966195 5843 ts_tablet_manager.cc:1367] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240507 08:12:15.966631 7418 leader_election.cc:288] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:15.970455 5843 raft_consensus.cc:348] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.970726 5843 raft_consensus.cc:374] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:15.970825 5843 raft_consensus.cc:729] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:15.971055 5843 consensus_queue.cc:260] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:15.971647 5843 ts_tablet_manager.cc:1398] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:15.971961 5843 tablet_bootstrap.cc:492] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:15.974224 5843 tablet_bootstrap.cc:654] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:15.975373 5841 ts_tablet_manager.cc:1875] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:15.948611 5259 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 4b2a2cb72f164e86ac5255dbd5be089e
I20240507 08:12:15.962749 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_638c97cc24874e01afa74a619d4b9dc4" } modify_external_catalogs: true
I20240507 08:12:15.978076 5578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.978251 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_638c97cc24874e01afa74a619d4b9dc4" } modify_external_catalogs: true
I20240507 08:12:15.979027 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 8cb968d91481464c830133873b16b8e0 (DEFAULT_TABLE table=default.Table_42a34eaa8a514f349f1a092abfff155f [id=abbc1d232ee54624a3cd2e6ca6e2433a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.979188 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 38fc0180e9d34d11885b819529407930 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:15.945555 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 33875331045146a9a02559a0b37d5c6f (table default.Table_c2c8b03ca2bf496d8112eb02078a24c4 [id=3273c97f16d6408f9c12e4a04711be33]) successfully deleted
I20240507 08:12:15.979568 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.979614 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet dc74342c8aee4017a212aa88a7f3dc06 (table default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]) successfully deleted
I20240507 08:12:15.979939 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 8eb5555330ea49a6b560e90d54ee5f5f (DEFAULT_TABLE table=default.Table_43a808c00e4b4c99b2a63595d71acbd4 [id=a88c66785dd04de5ac0150bf0aa7ae56]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.980257 5552 tablet_service.cc:1467] Processing CreateTablet for tablet eb1213d587ea47bfb55a6cfa413430e8 (DEFAULT_TABLE table=default.Table_44bcd391ca2f4d08a04fd38c02b6f957 [id=a1086570b84e4bfe9c28b02a0e4ca9bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.980621 5548 tablet_service.cc:1467] Processing CreateTablet for tablet a54aa68a2b364d5f8f4f4c217bb3e167 (DEFAULT_TABLE table=default.Table_0a81eec1a1d34fa8bdbe9fac1d360516 [id=8ae9a7147ada4165b5d6197138892556]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.980607 5427 tablet_service.cc:1467] Processing CreateTablet for tablet c33ba079a17b4eafb8031be383c91106 (DEFAULT_TABLE table=default.Table_5c436dbf3c0f4f879e1d3a57d6405594 [id=c6e925419297458087d106a24ce71803]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.980947 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ba079a17b4eafb8031be383c91106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.980913 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 33a58839996b414d805c6b2873254562 (DEFAULT_TABLE table=default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0 [id=c3ee217cbe574081a5ad8987ad9fc370]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.981215 5520 leader_election.cc:341] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e298f11ff9f54e48b8de435f6849f39a
I20240507 08:12:15.979400 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:15.978955 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 0ec8ef4c7f74420088cdb05294764b99 (DEFAULT_TABLE table=default.Table_47367ed796754576afd6b024153d847f [id=2fa2e4cc886c443fa9bb6cbe525e523c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.987183 5557 tablet_service.cc:1467] Processing CreateTablet for tablet 8868c2ff8b434200b427d4198b0374d9 (DEFAULT_TABLE table=default.Table_7c08a4452a324ca5b3cc664cfef163ad [id=a345eda955bd4c168843fae6c0eb9943]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.988759 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 33a58839996b414d805c6b2873254562 (DEFAULT_TABLE table=default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0 [id=c3ee217cbe574081a5ad8987ad9fc370]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.989190 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a58839996b414d805c6b2873254562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:15.991128 7079 ts_tablet_manager.cc:1888] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:15.993038 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394] alter complete (version 3)
I20240507 08:12:15.992651 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e14487e625374060b118c9cce4500b3f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:15.993425 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 818246c9039e4e00aed16701ab9b8271 (DEFAULT_TABLE table=default.Table_eb491699584f4d18ad9cb26ed87fd99b [id=9f57749f2e8e4390af59816f6ab59067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.993716 7079 log.cc:1196] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/35e34b60b78b402faf51e003c97db787
I20240507 08:12:15.993129 5546 tablet_service.cc:1467] Processing CreateTablet for tablet c33ba079a17b4eafb8031be383c91106 (DEFAULT_TABLE table=default.Table_5c436dbf3c0f4f879e1d3a57d6405594 [id=c6e925419297458087d106a24ce71803]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.994272 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee977e6c7034aaa8148ec6bcfeedae2 (DEFAULT_TABLE table=default.Table_40e14999737746f0984e6386706a3d9c [id=2bbd1eed5ca74f24bbb08b803e9816e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.994637 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 8b47c57b0f964c7894461a594bb22539 (DEFAULT_TABLE table=default.Table_1f3f2ed28af24116b1d81eecee74897c [id=aa491803c5144419aa6df4c2f0a7c8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.994982 5562 tablet_service.cc:1467] Processing CreateTablet for tablet fce52cf5871c402caeb769d1ca4c408a (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.995146 5551 tablet_service.cc:1467] Processing CreateTablet for tablet dd97542e3db0413c8272ff2971801be7 (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.995175 5656 consensus_peers.cc:485] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5471c74c33cd42e8aaf47f71f61adff7. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:15.995348 5558 tablet_service.cc:1467] Processing CreateTablet for tablet a1a06e10c76546b190adbc63820c080e (DEFAULT_TABLE table=default.Table_e7a8a764bfdf4f358f30d679d0bbf5d6 [id=dd79de02fb044d2cab9c8f11e64097b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.995434 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 919078225a45448a9b0013e340c041b9 (DEFAULT_TABLE table=default.Table_2915a6cc4171446abe445f6c4fb1eb44 [id=3c6228d5a39a41e78cbff2a8ee8e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.995622 5553 tablet_service.cc:1467] Processing CreateTablet for tablet c59886326a48434fb737976d9f571cd3 (DEFAULT_TABLE table=default.Table_2cd819bfa63841308bf0554cd623e25a [id=c6905d870b9b45308aaca4db0d9f34ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.995872 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf9b2427a24eaf9d9ab14a00bed5df (DEFAULT_TABLE table=default.Table_ba61f9a9c956447589c8e0db1eaa6f6f [id=5d80f7b5b3e7459692cbc3140f7822fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.994982 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 140b4601f1a043329c4de612a3ae244c (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:15.995010 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 7c73a5096ae149d6ba2b792f1d6966d5 (DEFAULT_TABLE table=default.Table_2f032a109d8d4902bd1611542a32e9d9 [id=c8bc7ac4fed1481ca19cee47f8310763]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:15.999437 5654 leader_election.cc:341] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:16.000111 5654 consensus_peers.cc:485] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.000643 5654 consensus_peers.cc:485] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.000752 5654 leader_election.cc:341] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:16.000977 5654 consensus_peers.cc:485] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.001689 6513 ts_tablet_manager.cc:1888] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.001848 6513 log.cc:1196] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b2df872d04b94c30872b03367df3cc31
I20240507 08:12:16.002257 6513 ts_tablet_manager.cc:1909] T b2df872d04b94c30872b03367df3cc31 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.002774 5843 tablet_bootstrap.cc:492] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:16.002920 5843 ts_tablet_manager.cc:1367] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.004s
W20240507 08:12:16.003808 5378 consensus_peers.cc:485] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.004107 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e298f11ff9f54e48b8de435f6849f39a" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:16.004349 5378 leader_election.cc:341] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.004447 5378 leader_election.cc:302] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.003790 5843 raft_consensus.cc:348] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:16.004758 5517 consensus_peers.cc:485] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 -> Peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Couldn't send request to peer 1cbb887175f345ada2433adaec906fd4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 79524df2415146ab926425c42068626c. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.004804 5843 raft_consensus.cc:374] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.004884 5843 raft_consensus.cc:729] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.004920 5517 leader_election.cc:302] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:16.004943 5378 leader_election.cc:341] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.005021 5378 leader_election.cc:302] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.005156 7418 raft_consensus.cc:2726] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.005141 5843 consensus_queue.cc:260] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:16.005352 5378 leader_election.cc:341] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.005432 5378 leader_election.cc:302] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.005650 7469 raft_consensus.cc:2726] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.005946 7469 raft_consensus.cc:2726] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.006143 7469 raft_consensus.cc:2726] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.006649 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8dc2055582bf4aa6aa58fa48b51f7b1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.007222 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc2055582bf4aa6aa58fa48b51f7b1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.009212 5654 consensus_peers.cc:485] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.010632 7079 ts_tablet_manager.cc:1909] T 35e34b60b78b402faf51e003c97db787 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:16.010957 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e77fae0cabcb447b9e8e6777828e20cf" } new_table_name: "default.Table_49409d424a624e8b91ff652bf16ad729" modify_external_catalogs: true
I20240507 08:12:16.012105 5841 ts_tablet_manager.cc:1888] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:16.012229 5841 log.cc:1196] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/ba7c1aaf55d74c979ec1fc0a57f15f44
I20240507 08:12:16.012575 5841 ts_tablet_manager.cc:1909] T ba7c1aaf55d74c979ec1fc0a57f15f44 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.012898 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1037) took 3748 ms. Trace:
I20240507 08:12:16.012984 5841 rpcz_store.cc:276] 0507 08:12:12.264517 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.264595 (+ 78us) service_pool.cc:225] Handling call
0507 08:12:15.954322 (+3689727us) ts_tablet_manager.cc:1120] Deleting tablet ba7c1aaf55d74c979ec1fc0a57f15f44
0507 08:12:15.954334 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.997170 (+ 42836us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.012004 (+ 14834us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.012879 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14647,"tablet-delete.queue_time_us":3689530}
I20240507 08:12:16.017221 5839 tablet_bootstrap.cc:492] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:16.017077 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_428bd11419b24adf8a02eb81b03d995f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.017354 5839 ts_tablet_manager.cc:1367] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.081s user 0.003s sys 0.000s
I20240507 08:12:16.017683 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1042) took 3449 ms. Trace:
I20240507 08:12:16.017758 5841 rpcz_store.cc:276] 0507 08:12:12.567740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.567864 (+ 124us) service_pool.cc:225] Handling call
0507 08:12:16.017529 (+3449665us) ts_tablet_manager.cc:1120] Deleting tablet 1578a63b4e3e4743b1057fa69653cbef
0507 08:12:16.017541 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.017665 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3426342}
I20240507 08:12:16.017766 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_454ae1a1af424a77a79ea0010b86a0bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.018078 5839 raft_consensus.cc:348] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:16.018208 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_454ae1a1af424a77a79ea0010b86a0bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.018285 5839 raft_consensus.cc:374] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.018337 5839 raft_consensus.cc:729] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.018558 5839 consensus_queue.cc:260] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.018648 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1041) took 3451 ms. Trace:
I20240507 08:12:16.018728 5841 rpcz_store.cc:276] 0507 08:12:12.567634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.567791 (+ 157us) service_pool.cc:225] Handling call
0507 08:12:16.018548 (+3450757us) ts_tablet_manager.cc:1120] Deleting tablet 2b14c80b96bb43088e1d1354a1dbdd31
0507 08:12:16.018557 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.018632 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3428769}
I20240507 08:12:16.019528 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1046) took 3358 ms. Trace:
I20240507 08:12:16.019975 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_63a2b5e8edd14d6aa283025fbf8df787" } new_table_name: "default.Table_b02b0407eb5145aebdb81d620532dc5e" modify_external_catalogs: true
I20240507 08:12:16.020116 5839 ts_tablet_manager.cc:1398] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240507 08:12:16.020298 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_428bd11419b24adf8a02eb81b03d995f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.023460 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 156673b3031d436db6391ff971641b84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.026203 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cb968d91481464c830133873b16b8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.031396 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5d7fffd25e7d4dc1824bd0769a8e9fab" } new_table_name: "default.Table_fd204b1d0b0943bcbf82f683d5a71256" modify_external_catalogs: true
I20240507 08:12:16.032737 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1acd7fcd5bbd40d3ad61d63adf97169e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.033188 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1acd7fcd5bbd40d3ad61d63adf97169e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.035368 5840 raft_consensus.cc:729] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.035715 5840 consensus_queue.cc:260] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.036571 5840 ts_tablet_manager.cc:1398] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.293s user 0.001s sys 0.001s
I20240507 08:12:16.036926 5840 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.037801 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eb5555330ea49a6b560e90d54ee5f5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.041446 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb1213d587ea47bfb55a6cfa413430e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.044945 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a54aa68a2b364d5f8f4f4c217bb3e167. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.045903 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8868c2ff8b434200b427d4198b0374d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.048818 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a58839996b414d805c6b2873254562. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.052214 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818246c9039e4e00aed16701ab9b8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.053805 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c33ba079a17b4eafb8031be383c91106. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.054783 5409 tablet_service.cc:1467] Processing CreateTablet for tablet a1a06e10c76546b190adbc63820c080e (DEFAULT_TABLE table=default.Table_e7a8a764bfdf4f358f30d679d0bbf5d6 [id=dd79de02fb044d2cab9c8f11e64097b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.055248 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a06e10c76546b190adbc63820c080e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.057138 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee977e6c7034aaa8148ec6bcfeedae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.059816 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 0baf9b2427a24eaf9d9ab14a00bed5df (DEFAULT_TABLE table=default.Table_ba61f9a9c956447589c8e0db1eaa6f6f [id=5d80f7b5b3e7459692cbc3140f7822fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.060205 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf9b2427a24eaf9d9ab14a00bed5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.061640 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1043) took 2679 ms. Trace:
I20240507 08:12:16.061731 7079 rpcz_store.cc:276] 0507 08:12:13.382282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.382339 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:15.732040 (+2349701us) ts_tablet_manager.cc:1120] Deleting tablet 35e34b60b78b402faf51e003c97db787
0507 08:12:15.732049 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.985499 (+253450us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.991036 (+ 5537us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.061617 (+ 70581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349515}
I20240507 08:12:16.062130 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1058) took 2458 ms. Trace:
I20240507 08:12:16.062206 7079 rpcz_store.cc:276] 0507 08:12:13.603707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.603771 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:16.061999 (+2458228us) ts_tablet_manager.cc:1120] Deleting tablet 497ab12edc724eef8d6e7b631662a4ab
0507 08:12:16.062009 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.062113 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2458051}
I20240507 08:12:16.062341 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b47c57b0f964c7894461a594bb22539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.062516 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1059) took 2454 ms. Trace:
I20240507 08:12:16.064981 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (table default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]) successfully deleted
W20240507 08:12:16.065271 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1578a63b4e3e4743b1057fa69653cbef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1578a63b4e3e4743b1057fa69653cbef already in progress: creating tablet
I20240507 08:12:16.065447 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c73a5096ae149d6ba2b792f1d6966d5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.065469 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2b14c80b96bb43088e1d1354a1dbdd31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b14c80b96bb43088e1d1354a1dbdd31 already in progress: creating tablet
I20240507 08:12:16.065845 5839 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:16.066334 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919078225a45448a9b0013e340c041b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.068157 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1a06e10c76546b190adbc63820c080e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.070755 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97542e3db0413c8272ff2971801be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.073642 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce52cf5871c402caeb769d1ca4c408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.076395 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0baf9b2427a24eaf9d9ab14a00bed5df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.079116 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ec8ef4c7f74420088cdb05294764b99. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.082386 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59886326a48434fb737976d9f571cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.082767 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140b4601f1a043329c4de612a3ae244c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.088088 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 818246c9039e4e00aed16701ab9b8271 (DEFAULT_TABLE table=default.Table_eb491699584f4d18ad9cb26ed87fd99b [id=9f57749f2e8e4390af59816f6ab59067]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.088474 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818246c9039e4e00aed16701ab9b8271. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.090590 5425 tablet_service.cc:1467] Processing CreateTablet for tablet c59886326a48434fb737976d9f571cd3 (DEFAULT_TABLE table=default.Table_2cd819bfa63841308bf0554cd623e25a [id=c6905d870b9b45308aaca4db0d9f34ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.090922 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59886326a48434fb737976d9f571cd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.092439 5413 tablet_service.cc:1467] Processing CreateTablet for tablet dd97542e3db0413c8272ff2971801be7 (DEFAULT_TABLE table=default.Table_84b116dd65724393b389f4455634ac10 [id=e9e4b4d2a9f140fd82519b16df2b4a65]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.092763 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd97542e3db0413c8272ff2971801be7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.092588 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0dbb7701a8a94990a7d51582422d59ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.093133 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dbb7701a8a94990a7d51582422d59ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.094995 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 8b47c57b0f964c7894461a594bb22539 (DEFAULT_TABLE table=default.Table_1f3f2ed28af24116b1d81eecee74897c [id=aa491803c5144419aa6df4c2f0a7c8d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.095306 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b47c57b0f964c7894461a594bb22539. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.095449 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_aac731cdd6804a3da20d6eae6ea0e45b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.095916 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aac731cdd6804a3da20d6eae6ea0e45b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.097826 5420 tablet_service.cc:1467] Processing CreateTablet for tablet fce52cf5871c402caeb769d1ca4c408a (DEFAULT_TABLE table=default.Table_cdd1eb0440bf4b9fbf81d06633d206eb [id=2df5eb5d73904953a793fd50feb4cd34]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.098131 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fce52cf5871c402caeb769d1ca4c408a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.101429 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_269bbea2c2e04830a74ce31662db3c24" } new_table_name: "default.Table_9996de3de49247a88805d3d7fcc98613" modify_external_catalogs: true
I20240507 08:12:16.101842 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 919078225a45448a9b0013e340c041b9 (DEFAULT_TABLE table=default.Table_2915a6cc4171446abe445f6c4fb1eb44 [id=3c6228d5a39a41e78cbff2a8ee8e65af]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.102149 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919078225a45448a9b0013e340c041b9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.019873 5841 rpcz_store.cc:276] 0507 08:12:12.661020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.661081 (+ 61us) service_pool.cc:225] Handling call
0507 08:12:16.019437 (+3358356us) ts_tablet_manager.cc:1120] Deleting tablet 5855bc973d354a69a0579d9a76b3f92b
0507 08:12:16.019445 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.019514 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226192}
I20240507 08:12:16.113293 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1049) took 3232 ms. Trace:
I20240507 08:12:16.113381 5841 rpcz_store.cc:276] 0507 08:12:12.880708 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.880747 (+ 39us) service_pool.cc:225] Handling call
0507 08:12:16.113177 (+3232430us) ts_tablet_manager.cc:1120] Deleting tablet a3535f1fbfe34f6e95e9895771ae1350
0507 08:12:16.113186 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.113274 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3232192}
I20240507 08:12:16.113818 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 140b4601f1a043329c4de612a3ae244c (DEFAULT_TABLE table=default.Table_c8c312b582bf4a4ab5cd690c377c47f5 [id=557386b6c3114868bdd417b46b90e2c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.114250 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 140b4601f1a043329c4de612a3ae244c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.124084 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1050) took 3243 ms. Trace:
I20240507 08:12:16.124187 5841 rpcz_store.cc:276] 0507 08:12:12.880797 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.880847 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:16.123962 (+3243115us) ts_tablet_manager.cc:1120] Deleting tablet c1932ffe7d37455c8aa6bb83ffd11498
0507 08:12:16.123972 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.124064 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46,"tablet-delete.queue_time_us":3242971}
I20240507 08:12:16.124529 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1048) took 3243 ms. Trace:
I20240507 08:12:16.124601 5841 rpcz_store.cc:276] 0507 08:12:12.880619 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.894232 (+ 13613us) service_pool.cc:225] Handling call
0507 08:12:16.124427 (+3230195us) ts_tablet_manager.cc:1120] Deleting tablet 37a23d112fcd4ea1b0c2f86133e82e32
0507 08:12:16.124438 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.124514 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3222296}
I20240507 08:12:16.124850 5841 tablet_replica.cc:330] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.125017 5841 raft_consensus.cc:2229] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.125344 5841 raft_consensus.cc:2258] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.125902 5841 ts_tablet_manager.cc:1875] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:16.127002 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 636e8f123b9540f3afedf02e4df5dfb5 consensus state has no leader
W20240507 08:12:16.128425 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0929c348711341d9b3529f22ff2b3df9 consensus state has no leader
I20240507 08:12:16.129535 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 7c73a5096ae149d6ba2b792f1d6966d5 (DEFAULT_TABLE table=default.Table_2f032a109d8d4902bd1611542a32e9d9 [id=c8bc7ac4fed1481ca19cee47f8310763]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.129947 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c73a5096ae149d6ba2b792f1d6966d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.132246 5841 ts_tablet_manager.cc:1888] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:16.132354 5841 log.cc:1196] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/34adee00b529424b9f274c61ce73b0bc
I20240507 08:12:16.132697 5841 ts_tablet_manager.cc:1909] T 34adee00b529424b9f274c61ce73b0bc P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.132977 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1074) took 2796 ms. Trace:
I20240507 08:12:16.133054 5841 rpcz_store.cc:276] 0507 08:12:13.336301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.354536 (+ 18235us) service_pool.cc:225] Handling call
0507 08:12:16.124809 (+2770273us) ts_tablet_manager.cc:1120] Deleting tablet 34adee00b529424b9f274c61ce73b0bc
0507 08:12:16.124817 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.128592 (+ 3775us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.132170 (+ 3578us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.132960 (+ 790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2731912}
I20240507 08:12:16.133314 5841 tablet_replica.cc:330] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.133834 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 (DEFAULT_TABLE table=default.Table_091c1a53ece34eb3bbc8265fa45f9088 [id=e48fba499b3f47728a1a2fc848800e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.134338 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42f6e958dd94e3cbecd5c4bd6a6d829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.135535 5565 tablet_service.cc:1467] Processing CreateTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 (DEFAULT_TABLE table=default.Table_091c1a53ece34eb3bbc8265fa45f9088 [id=e48fba499b3f47728a1a2fc848800e36]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.136040 5412 tablet_service.cc:1467] Processing CreateTablet for tablet d4216050db0146e59aac37b6e8b0018c (DEFAULT_TABLE table=default.Table_7a8bd4e6c9d149189ae4d0f6877c8cdd [id=2c7582e1958e4126b24894a136730344]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.136338 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4216050db0146e59aac37b6e8b0018c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.137849 5413 tablet_service.cc:1467] Processing CreateTablet for tablet b967f56b73aa4c4eb838c42c831b9712 (DEFAULT_TABLE table=default.Table_f2867d292c9d45a3b77c426279d29c32 [id=361b5acb391340c6a150f3f081cfedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.138149 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967f56b73aa4c4eb838c42c831b9712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.138090 5418 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ece8042754f4e9877f286f02ffd64 (DEFAULT_TABLE table=default.Table_ca9f79f6fd724517a82be3b0bf66dcf8 [id=e91aa5cee994450d8a06443bfc049be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.138363 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ece8042754f4e9877f286f02ffd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.140583 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 67d479e3dd17488e914adf01e630169e (DEFAULT_TABLE table=default.Table_7d3cd26b01594f059ae92b67fe87d846 [id=74cae62c166e44339803e2700dc5da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.141068 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d479e3dd17488e914adf01e630169e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.141446 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b1919d9d46d884c5784accbb5304 (DEFAULT_TABLE table=default.Table_087f9c936f35417f854ffeeb097ca835 [id=3b47c5283e8f42a59dfea9fea27d08e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.145625 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 80d3b1919d9d46d884c5784accbb5304 (DEFAULT_TABLE table=default.Table_087f9c936f35417f854ffeeb097ca835 [id=3b47c5283e8f42a59dfea9fea27d08e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.146129 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b1919d9d46d884c5784accbb5304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.039546 5840 tablet_bootstrap.cc:654] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.088063 7101 consensus_queue.cc:1035] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.073s
W20240507 08:12:16.115018 5654 consensus_peers.cc:485] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.161991 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 47026.8us
W20240507 08:12:16.124220 5656 consensus_peers.cc:485] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33875331045146a9a02559a0b37d5c6f. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.162539 5654 consensus_peers.cc:485] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.162858 5654 consensus_peers.cc:485] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.163216 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2a9bf32660374d72a70a5c5b228881a1" } new_table_name: "default.Table_d2f247862fcb486e9776cf2e1b2107bb" modify_external_catalogs: true
I20240507 08:12:16.070379 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1006) took 4836 ms. Trace:
I20240507 08:12:16.164011 6513 rpcz_store.cc:276] 0507 08:12:11.234182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.234666 (+ 484us) service_pool.cc:225] Handling call
0507 08:12:15.659923 (+4425257us) ts_tablet_manager.cc:1120] Deleting tablet b2df872d04b94c30872b03367df3cc31
0507 08:12:15.659932 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:15.724811 (+ 64879us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:15.961921 (+237110us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.070350 (+108429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4425086}
I20240507 08:12:16.164633 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b2df872d04b94c30872b03367df3cc31 (table default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]) successfully deleted
I20240507 08:12:16.166185 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5e6f4cb669d74f3594bba624c848009d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.166715 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e6f4cb669d74f3594bba624c848009d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.166855 6513 tablet_replica.cc:330] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.167112 6513 raft_consensus.cc:2229] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.167436 6513 raft_consensus.cc:2258] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.168615 6513 ts_tablet_manager.cc:1875] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.169247 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_816c7491958e48f29ae98e8b458aa440" } modify_external_catalogs: true
I20240507 08:12:16.169416 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_816c7491958e48f29ae98e8b458aa440" } modify_external_catalogs: true
I20240507 08:12:16.169889 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6500147915384d518c459e558a9ce1d6" } modify_external_catalogs: true
I20240507 08:12:16.170022 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6500147915384d518c459e558a9ce1d6" } modify_external_catalogs: true
I20240507 08:12:16.170563 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.173105 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.175551 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.175834 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80bfaa8d097646e3bcce7022f314f4fa consensus state has no leader
I20240507 08:12:16.176208 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bc1f5c0697f04fbba453282facb1a6a1" } modify_external_catalogs: true
I20240507 08:12:16.176425 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bc1f5c0697f04fbba453282facb1a6a1" } modify_external_catalogs: true
I20240507 08:12:16.177033 5424 tablet_service.cc:1467] Processing CreateTablet for tablet f1980322500f42b99180b86cc259ca3f (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.177424 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980322500f42b99180b86cc259ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.180198 5413 tablet_service.cc:1514] Processing DeleteTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.181921 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 9ee977e6c7034aaa8148ec6bcfeedae2 (DEFAULT_TABLE table=default.Table_40e14999737746f0984e6386706a3d9c [id=2bbd1eed5ca74f24bbb08b803e9816e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.182499 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ee977e6c7034aaa8148ec6bcfeedae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.183679 7100 consensus_queue.cc:1035] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:16.189318 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5855bc973d354a69a0579d9a76b3f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5855bc973d354a69a0579d9a76b3f92b already in progress: creating tablet
I20240507 08:12:16.191533 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 35e34b60b78b402faf51e003c97db787 (table default.Table_7007a63c36ea47c78769b5bcafec4cf9 [id=71fd821e13734824a24f76fa2c220f1a]) successfully deleted
W20240507 08:12:16.192065 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 497ab12edc724eef8d6e7b631662a4ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 497ab12edc724eef8d6e7b631662a4ab already in progress: creating tablet
W20240507 08:12:16.198596 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a3535f1fbfe34f6e95e9895771ae1350 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3535f1fbfe34f6e95e9895771ae1350 already in progress: creating tablet
W20240507 08:12:16.199016 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c1932ffe7d37455c8aa6bb83ffd11498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1932ffe7d37455c8aa6bb83ffd11498 already in progress: creating tablet
W20240507 08:12:16.199295 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 37a23d112fcd4ea1b0c2f86133e82e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a23d112fcd4ea1b0c2f86133e82e32 already in progress: creating tablet
I20240507 08:12:16.199556 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]) successfully deleted
I20240507 08:12:16.205227 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_edb3317ccbd440c99457c0d12ce3c25c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.206302 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edb3317ccbd440c99457c0d12ce3c25c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.206933 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_50f7ec043384417799974a1e441c4747"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.207353 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50f7ec043384417799974a1e441c4747 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.209532 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 379807692b6147599ed71578869a52e2 consensus state has no leader
I20240507 08:12:16.209944 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f9c3b5637d424b1199065768db5c961a" } modify_external_catalogs: true
I20240507 08:12:16.210125 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f9c3b5637d424b1199065768db5c961a" } modify_external_catalogs: true
I20240507 08:12:16.211164 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:16.217703 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.130s user 0.001s sys 0.000s
I20240507 08:12:16.151311 5554 tablet_service.cc:1467] Processing CreateTablet for tablet d4216050db0146e59aac37b6e8b0018c (DEFAULT_TABLE table=default.Table_7a8bd4e6c9d149189ae4d0f6877c8cdd [id=2c7582e1958e4126b24894a136730344]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.153909 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 67d479e3dd17488e914adf01e630169e (DEFAULT_TABLE table=default.Table_7d3cd26b01594f059ae92b67fe87d846 [id=74cae62c166e44339803e2700dc5da6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.162153 7079 rpcz_store.cc:276] 0507 08:12:13.608338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.608393 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:16.062419 (+2454026us) ts_tablet_manager.cc:1120] Deleting tablet 52e9c1158e27465280e1043d91917e3e
0507 08:12:16.062427 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.062502 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2453853}
I20240507 08:12:16.236045 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1060) took 2608 ms. Trace:
I20240507 08:12:16.236137 7079 rpcz_store.cc:276] 0507 08:12:13.627730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.627791 (+ 61us) service_pool.cc:225] Handling call
0507 08:12:16.235933 (+2608142us) ts_tablet_manager.cc:1120] Deleting tablet 74c9c649ec3a4a3db414b78ef9792423
0507 08:12:16.235940 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.236028 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2607969}
I20240507 08:12:16.236505 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1063) took 2582 ms. Trace:
I20240507 08:12:16.236589 7079 rpcz_store.cc:276] 0507 08:12:13.654527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.654582 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:16.236408 (+2581826us) ts_tablet_manager.cc:1120] Deleting tablet 9cf92632da4643cfab913cd925fccef9
0507 08:12:16.236416 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.236492 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2581662}
I20240507 08:12:16.236896 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1066) took 2553 ms. Trace:
I20240507 08:12:16.236987 7079 rpcz_store.cc:276] 0507 08:12:13.683471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.683519 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:16.236803 (+2553284us) ts_tablet_manager.cc:1120] Deleting tablet c8aaabc3e26140fda6f86ce0e518b8f5
0507 08:12:16.236811 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.236880 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2553121}
W20240507 08:12:16.237128 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 52e9c1158e27465280e1043d91917e3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52e9c1158e27465280e1043d91917e3e already in progress: creating tablet
I20240507 08:12:16.237287 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1070) took 2492 ms. Trace:
I20240507 08:12:16.237361 7079 rpcz_store.cc:276] 0507 08:12:13.745021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.745080 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:16.237196 (+2492116us) ts_tablet_manager.cc:1120] Deleting tablet d07221497ce7468fb86c257492cef397
0507 08:12:16.237205 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.237274 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2491904}
W20240507 08:12:16.237481 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 74c9c649ec3a4a3db414b78ef9792423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c9c649ec3a4a3db414b78ef9792423 already in progress: creating tablet
I20240507 08:12:16.237689 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1080) took 2380 ms. Trace:
I20240507 08:12:16.237787 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b2df872d04b94c30872b03367df3cc31 with delete_type TABLET_DATA_DELETED (replaced by tablet 4e37b73134cf42ad84b6eec90c9b805f) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.238623 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 994f50ea19a2445288663b7d7b7ad291 (DEFAULT_TABLE table=default.Table_bc1f5c0697f04fbba453282facb1a6a1 [id=41cc0579a0124faa915d338c27ba480a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:16.238852 5656 consensus_peers.cc:485] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc74342c8aee4017a212aa88a7f3dc06. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.239194 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994f50ea19a2445288663b7d7b7ad291. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.239362 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9cf92632da4643cfab913cd925fccef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf92632da4643cfab913cd925fccef9 already in progress: creating tablet
W20240507 08:12:16.239534 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c8aaabc3e26140fda6f86ce0e518b8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8aaabc3e26140fda6f86ce0e518b8f5 already in progress: creating tablet
W20240507 08:12:16.239698 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d07221497ce7468fb86c257492cef397 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d07221497ce7468fb86c257492cef397 already in progress: creating tablet
I20240507 08:12:16.237766 7079 rpcz_store.cc:276] 0507 08:12:13.857486 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.857532 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:16.237567 (+2380035us) ts_tablet_manager.cc:1120] Deleting tablet 4c62499ccee34036b23e4fe0804fd2ae
0507 08:12:16.237591 (+ 24us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.237670 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2379834}
I20240507 08:12:16.241719 7079 tablet_replica.cc:330] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:16.241873 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4c62499ccee34036b23e4fe0804fd2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c62499ccee34036b23e4fe0804fd2ae already in progress: creating tablet
I20240507 08:12:16.241907 7079 raft_consensus.cc:2229] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.242048 7079 raft_consensus.cc:2258] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.242551 7079 ts_tablet_manager.cc:1875] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.189776 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 994f50ea19a2445288663b7d7b7ad291 (DEFAULT_TABLE table=default.Table_bc1f5c0697f04fbba453282facb1a6a1 [id=41cc0579a0124faa915d338c27ba480a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.215085 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c42f6e958dd94e3cbecd5c4bd6a6d829. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.246857 5656 consensus_peers.cc:485] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 85dc58133d2b4f5496b4e570ae6494c0. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.247066 7378 raft_consensus.cc:482] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.247188 7378 raft_consensus.cc:504] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.247735 7378 leader_election.cc:288] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.249032 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385f4c1eb7f4493a924b813734ddcfdb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:16.249298 5720 raft_consensus.cc:2454] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:16.250121 5582 raft_consensus.cc:3037] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.251538 5377 leader_election.cc:302] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:16.251821 5414 tablet_service.cc:1467] Processing CreateTablet for tablet f921a7e89a874c64817bff535ad4eda7 (DEFAULT_TABLE table=default.Table_97f7aa309ac14b62b69cbe42ce8ff48b [id=947babe79251496d855060b66eecef30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.252183 7378 raft_consensus.cc:2781] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:16.252254 7378 raft_consensus.cc:482] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:16.252305 7378 raft_consensus.cc:3037] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.255370 7378 raft_consensus.cc:504] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.256393 7378 leader_election.cc:288] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.256685 7100 consensus_queue.cc:1035] T 3b9429e5cc274198974b27c40c7e4e37 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:16.259341 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80d3b1919d9d46d884c5784accbb5304. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.259380 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385f4c1eb7f4493a924b813734ddcfdb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:16.259617 5720 raft_consensus.cc:3037] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.259793 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4216050db0146e59aac37b6e8b0018c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.260116 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67d479e3dd17488e914adf01e630169e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.224581 5546 tablet_service.cc:1467] Processing CreateTablet for tablet fc4ece8042754f4e9877f286f02ffd64 (DEFAULT_TABLE table=default.Table_ca9f79f6fd724517a82be3b0bf66dcf8 [id=e91aa5cee994450d8a06443bfc049be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.227823 5560 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ddb7dd28f43488581d3a75bf1d68a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.230262 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.230780 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.231786 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b4c69d38e484f5c819b7dd915bf4410 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.232929 5561 tablet_service.cc:1467] Processing CreateTablet for tablet f1980322500f42b99180b86cc259ca3f (DEFAULT_TABLE table=default.Table_2aaf5f336efc491a8eacf261247f65bd [id=484c5d211a9a4df7939776303e1553c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.145535 5562 tablet_service.cc:1467] Processing CreateTablet for tablet b967f56b73aa4c4eb838c42c831b9712 (DEFAULT_TABLE table=default.Table_f2867d292c9d45a3b77c426279d29c32 [id=361b5acb391340c6a150f3f081cfedc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.261821 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f921a7e89a874c64817bff535ad4eda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.263702 5720 raft_consensus.cc:2454] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:16.264117 5377 leader_election.cc:302] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:16.264554 7378 raft_consensus.cc:2781] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:16.264847 7378 raft_consensus.cc:686] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:16.264693 7079 ts_tablet_manager.cc:1888] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:16.265213 7378 consensus_queue.cc:237] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.265287 7079 log.cc:1196] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/753019050fcf4dc3b69722c3eb475037
I20240507 08:12:16.269150 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b967f56b73aa4c4eb838c42c831b9712. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.278388 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b8c6f552832a45c4a2e28f40797db8fa
W20240507 08:12:16.278800 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04a14cf04d7a4ce3a2df939d0db515b4 consensus state has no leader
I20240507 08:12:16.279172 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6cc465fb947d406f85b3779471ec43cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.279248 7469 consensus_queue.cc:1035] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:16.279696 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cc465fb947d406f85b3779471ec43cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.281625 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c6f552832a45c4a2e28f40797db8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.282657 7469 consensus_queue.cc:1035] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:16.283877 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1980322500f42b99180b86cc259ca3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.288674 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc4ece8042754f4e9877f286f02ffd64. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.298669 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 994f50ea19a2445288663b7d7b7ad291. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.301685 6675 consensus_queue.cc:1035] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:16.306941 5437 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.053s sys 0.103s
W20240507 08:12:16.307627 5437 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.053s sys 0.103s
I20240507 08:12:16.312105 7408 raft_consensus.cc:482] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.312157 5841 raft_consensus.cc:2229] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.312229 7408 raft_consensus.cc:504] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.312925 7408 leader_election.cc:288] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.313692 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:16.314246 5377 leader_election.cc:341] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.315047 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8bb62a72ed2248f6886823b9c64ff4c6
I20240507 08:12:16.315289 5840 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.315547 5840 ts_tablet_manager.cc:1367] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.279s user 0.004s sys 0.000s
I20240507 08:12:16.316558 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.317523 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.318390 5840 raft_consensus.cc:348] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.318737 5840 raft_consensus.cc:374] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.318857 5840 raft_consensus.cc:729] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.319212 6675 raft_consensus.cc:482] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.319170 5840 consensus_queue.cc:260] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.319307 6675 raft_consensus.cc:504] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.319684 5840 ts_tablet_manager.cc:1398] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:16.319841 6675 leader_election.cc:288] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.319932 5840 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.323060 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c6f552832a45c4a2e28f40797db8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.328858 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839c8a5041f4c3da9bc0749afee1413" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:16.329097 5447 raft_consensus.cc:2454] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:16.329545 5656 leader_election.cc:302] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:16.329883 6675 raft_consensus.cc:2781] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:16.330009 6675 raft_consensus.cc:482] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:16.330081 6675 raft_consensus.cc:3037] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.333353 6675 raft_consensus.cc:504] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.334133 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_47712cc9868f40f9b5de852407b36598" } new_table_name: "default.Table_840fddb88c744ccc927287f323d16cc8" modify_external_catalogs: true
I20240507 08:12:16.334640 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839c8a5041f4c3da9bc0749afee1413" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:16.334806 5447 raft_consensus.cc:3037] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.334131 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_442b2e36429540cf9ea03222ff0a0ced"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.336390 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_442b2e36429540cf9ea03222ff0a0ced in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.336483 6675 leader_election.cc:288] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.338944 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839c8a5041f4c3da9bc0749afee1413" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:16.339334 7079 ts_tablet_manager.cc:1909] T 753019050fcf4dc3b69722c3eb475037 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:16.339623 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1083) took 2374 ms. Trace:
I20240507 08:12:16.339723 7079 rpcz_store.cc:276] 0507 08:12:13.965174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.968371 (+ 3197us) service_pool.cc:225] Handling call
0507 08:12:16.241674 (+2273303us) ts_tablet_manager.cc:1120] Deleting tablet 753019050fcf4dc3b69722c3eb475037
0507 08:12:16.241685 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.248209 (+ 6524us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.264191 (+ 15982us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.339610 (+ 75419us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2273079}
I20240507 08:12:16.338956 5555 tablet_service.cc:1467] Processing CreateTablet for tablet f921a7e89a874c64817bff535ad4eda7 (DEFAULT_TABLE table=default.Table_97f7aa309ac14b62b69cbe42ce8ff48b [id=947babe79251496d855060b66eecef30]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.340058 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1087) took 2083 ms. Trace:
I20240507 08:12:16.340188 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f921a7e89a874c64817bff535ad4eda7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.340124 7079 rpcz_store.cc:276] 0507 08:12:14.256315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.258986 (+ 2671us) service_pool.cc:225] Handling call
0507 08:12:16.339960 (+2080974us) ts_tablet_manager.cc:1120] Deleting tablet 7e26c7ef6f2845cabe09d09045d4feda
0507 08:12:16.339969 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.340045 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2080755}
I20240507 08:12:16.340451 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1086) took 2084 ms. Trace:
I20240507 08:12:16.340513 7079 rpcz_store.cc:276] 0507 08:12:14.256223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.304702 (+ 48479us) service_pool.cc:225] Handling call
0507 08:12:16.340361 (+2035659us) ts_tablet_manager.cc:1120] Deleting tablet 2d913dc032b44a4fbab59b42efcba58c
0507 08:12:16.340368 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.340437 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035413}
I20240507 08:12:16.340732 6513 ts_tablet_manager.cc:1888] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.340790 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1088) took 2060 ms. Trace:
I20240507 08:12:16.340855 6513 log.cc:1196] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e298f11ff9f54e48b8de435f6849f39a
I20240507 08:12:16.340847 7079 rpcz_store.cc:276] 0507 08:12:14.280745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.310584 (+ 29839us) service_pool.cc:225] Handling call
0507 08:12:16.340701 (+2030117us) ts_tablet_manager.cc:1120] Deleting tablet 11e779c5c97c44e5bcc7c0eb0194fd95
0507 08:12:16.340708 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.340777 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029928}
I20240507 08:12:16.341038 7079 tablet_replica.cc:330] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:16.341210 7079 raft_consensus.cc:2229] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.341244 6513 ts_tablet_manager.cc:1909] T e298f11ff9f54e48b8de435f6849f39a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.341353 7079 raft_consensus.cc:2258] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.341640 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1009) took 5107 ms. Trace:
I20240507 08:12:16.343945 7079 ts_tablet_manager.cc:1875] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.344498 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet b15c8ad2a8324c75a1ef49e219b962cf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.346838 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bdafa405a484456fa07e870095e4daa8" } new_table_name: "default.Table_7568e0a7af0845dc863059434b0ed0c1" modify_external_catalogs: true
I20240507 08:12:16.348263 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet 43644b4405f44bfd9a1afcaedbd68db5 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:16.350054 5654 leader_election.cc:341] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.351464 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7949df2d12e8461292e82198f29d9367"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.352094 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7949df2d12e8461292e82198f29d9367 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.357407 5447 raft_consensus.cc:2454] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:16.358002 5656 leader_election.cc:302] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:16.358289 6675 raft_consensus.cc:2781] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:16.358384 6675 raft_consensus.cc:686] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:16.358626 6675 consensus_queue.cc:237] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.343564 7418 raft_consensus.cc:482] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.358956 7418 raft_consensus.cc:504] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.359575 7418 leader_election.cc:288] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.359980 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c839c8a5041f4c3da9bc0749afee1413" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:16.360452 5520 leader_election.cc:341] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 6b4c69d38e484f5c819b7dd915bf4410
I20240507 08:12:16.362452 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.362417 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b4c69d38e484f5c819b7dd915bf4410" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:16.362754 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 753019050fcf4dc3b69722c3eb475037 (table default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]) successfully deleted
W20240507 08:12:16.362915 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.141s user 0.004s sys 0.000s
W20240507 08:12:16.363197 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 7e26c7ef6f2845cabe09d09045d4feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e26c7ef6f2845cabe09d09045d4feda already in progress: creating tablet
W20240507 08:12:16.363502 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 2d913dc032b44a4fbab59b42efcba58c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d913dc032b44a4fbab59b42efcba58c already in progress: creating tablet
W20240507 08:12:16.363785 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e779c5c97c44e5bcc7c0eb0194fd95 already in progress: creating tablet
I20240507 08:12:16.363989 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c6f552832a45c4a2e28f40797db8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.364138 5841 pending_rounds.cc:62] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:16.364213 5841 pending_rounds.cc:68] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924812589277184 op_type: NO_OP noop_request { }
I20240507 08:12:16.364336 5841 raft_consensus.cc:2866] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:16.364437 5841 raft_consensus.cc:2258] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.364369 5517 leader_election.cc:302] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:16.364766 7418 raft_consensus.cc:2726] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:16.365504 5654 leader_election.cc:341] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.367179 5840 tablet_bootstrap.cc:654] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:16.367925 7479 consensus_queue.cc:1159] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "dc665657d5b34fd7aec884e0f2668e11" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20240507 08:12:16.369447 5841 ts_tablet_manager.cc:1875] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.378088 5841 ts_tablet_manager.cc:1888] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.378274 5841 log.cc:1196] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:16.378696 5841 ts_tablet_manager.cc:1909] T 767905763ba14cb8bbc52151ae9f17e6 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.379099 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1080) took 2883 ms. Trace:
W20240507 08:12:16.380347 5656 consensus_peers.cc:485] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.380646 7079 ts_tablet_manager.cc:1888] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:16.380908 7079 log.cc:1196] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/364903504aa14d08b050a0cf83b82c71
I20240507 08:12:16.382982 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a00f64d4fd4146a4bad06d8aa738d872"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.383643 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a00f64d4fd4146a4bad06d8aa738d872 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.341732 6513 rpcz_store.cc:276] 0507 08:12:11.234537 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.234758 (+ 221us) service_pool.cc:225] Handling call
0507 08:12:16.166786 (+4932028us) ts_tablet_manager.cc:1120] Deleting tablet e298f11ff9f54e48b8de435f6849f39a
0507 08:12:16.166803 (+ 17us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.172916 (+ 6113us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.340602 (+167686us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.341614 (+ 1012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4931891}
I20240507 08:12:16.385394 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1007) took 5151 ms. Trace:
I20240507 08:12:16.385535 6513 rpcz_store.cc:276] 0507 08:12:11.234293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.234869 (+ 576us) service_pool.cc:225] Handling call
0507 08:12:16.385281 (+5150412us) ts_tablet_manager.cc:1120] Deleting tablet 168e49ae24314a7aae2291485ddeda44
0507 08:12:16.385293 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.385375 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5150265}
I20240507 08:12:16.386664 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e298f11ff9f54e48b8de435f6849f39a (table default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]) successfully deleted
W20240507 08:12:16.387063 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 168e49ae24314a7aae2291485ddeda44 because the tablet was not found. No further retry: Not found: Tablet not found: 168e49ae24314a7aae2291485ddeda44
I20240507 08:12:16.388906 5840 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.389122 5840 ts_tablet_manager.cc:1367] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.000s
I20240507 08:12:16.379382 5841 rpcz_store.cc:276] 0507 08:12:13.495415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.495469 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:16.133278 (+2637809us) ts_tablet_manager.cc:1120] Deleting tablet 767905763ba14cb8bbc52151ae9f17e6
0507 08:12:16.133285 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.364042 (+230757us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x616000213a54. stack: 00007fcb0a83a756 00007fcb0a83a510 00007fcb0986d48d 0000000000955804 00000000009557cd 000000000095519a 00007fcb160a8a00 00007fcb17382b83 00007fcb19e164ec 00007fcb19e48844 00007fcb19e29b30 00007fcb19e298bd 0000000000a60bcc 00007fcb0a880b74 00007fcb0a88419b 00007fcb0a884018
0507 08:12:16.372564 (+ 8522us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.378004 (+ 5440us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.379080 (+ 1076us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":410894208,"tablet-delete.queue_time_us":2637642}
I20240507 08:12:16.390086 5840 raft_consensus.cc:348] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.391448 5840 raft_consensus.cc:374] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.391561 5840 raft_consensus.cc:729] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.392241 5840 consensus_queue.cc:260] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.392832 5840 ts_tablet_manager.cc:1398] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:16.392966 7079 ts_tablet_manager.cc:1909] T 364903504aa14d08b050a0cf83b82c71 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:16.393193 5840 tablet_bootstrap.cc:492] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.385953 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1010) took 5151 ms. Trace:
I20240507 08:12:16.393756 6513 rpcz_store.cc:276] 0507 08:12:11.234620 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.235041 (+ 421us) service_pool.cc:225] Handling call
0507 08:12:16.385864 (+5150823us) ts_tablet_manager.cc:1120] Deleting tablet b7947ca2d8324dbb9c909ec50852fed8
0507 08:12:16.385873 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.385935 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5150699}
I20240507 08:12:16.394212 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1018) took 5015 ms. Trace:
I20240507 08:12:16.394340 6513 rpcz_store.cc:276] 0507 08:12:11.378968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.379026 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:16.394081 (+5015055us) ts_tablet_manager.cc:1120] Deleting tablet a6c50d46db8949ee978b87f83a640c0a
0507 08:12:16.394091 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.394196 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5014876}
I20240507 08:12:16.394708 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1019) took 5015 ms. Trace:
I20240507 08:12:16.394805 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1089) took 2063 ms. Trace:
I20240507 08:12:16.394827 6513 rpcz_store.cc:276] 0507 08:12:11.379086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.379357 (+ 271us) service_pool.cc:225] Handling call
0507 08:12:16.394619 (+5015262us) ts_tablet_manager.cc:1120] Deleting tablet a903c17e01c241bcb13787ef551395d2
0507 08:12:16.394627 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.394695 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5015055}
I20240507 08:12:16.395185 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1020) took 5015 ms. Trace:
I20240507 08:12:16.395047 7079 rpcz_store.cc:276] 0507 08:12:14.331384 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.337722 (+ 6338us) service_pool.cc:225] Handling call
0507 08:12:16.341005 (+2003283us) ts_tablet_manager.cc:1120] Deleting tablet 364903504aa14d08b050a0cf83b82c71
0507 08:12:16.341012 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.352436 (+ 11424us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.380516 (+ 28080us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.394757 (+ 14241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2003068}
I20240507 08:12:16.395292 6513 rpcz_store.cc:276] 0507 08:12:11.379727 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.382605 (+ 2878us) service_pool.cc:225] Handling call
0507 08:12:16.395094 (+5012489us) ts_tablet_manager.cc:1120] Deleting tablet e561c811a1de48f9bc91175a3d22805a
0507 08:12:16.395102 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.395171 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5012262}
W20240507 08:12:16.395597 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b7947ca2d8324dbb9c909ec50852fed8 because the tablet was not found. No further retry: Not found: Tablet not found: b7947ca2d8324dbb9c909ec50852fed8
W20240507 08:12:16.395925 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a6c50d46db8949ee978b87f83a640c0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6c50d46db8949ee978b87f83a640c0a already in progress: creating tablet
W20240507 08:12:16.396219 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a903c17e01c241bcb13787ef551395d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a903c17e01c241bcb13787ef551395d2 already in progress: creating tablet
W20240507 08:12:16.396507 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e561c811a1de48f9bc91175a3d22805a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e561c811a1de48f9bc91175a3d22805a already in progress: creating tablet
I20240507 08:12:16.396375 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1090) took 2041 ms. Trace:
I20240507 08:12:16.396937 6513 tablet_replica.cc:330] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.397195 6513 raft_consensus.cc:2229] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.397629 6513 raft_consensus.cc:2258] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.398197 6513 ts_tablet_manager.cc:1875] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.404611 6513 ts_tablet_manager.cc:1888] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.404808 6513 log.cc:1196] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/049483b0533642f2b46d8fa0e9b123cc
I20240507 08:12:16.405262 6513 ts_tablet_manager.cc:1909] T 049483b0533642f2b46d8fa0e9b123cc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.405651 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1021) took 5005 ms. Trace:
I20240507 08:12:16.405781 6513 rpcz_store.cc:276] 0507 08:12:11.400010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.400077 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:16.396889 (+4996812us) ts_tablet_manager.cc:1120] Deleting tablet 049483b0533642f2b46d8fa0e9b123cc
0507 08:12:16.396898 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.401295 (+ 4397us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.404524 (+ 3229us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.405635 (+ 1111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4996557}
I20240507 08:12:16.406193 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1023) took 5005 ms. Trace:
I20240507 08:12:16.406482 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 049483b0533642f2b46d8fa0e9b123cc (table default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]) successfully deleted
I20240507 08:12:16.406785 6513 rpcz_store.cc:276] 0507 08:12:11.400446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.400530 (+ 84us) service_pool.cc:225] Handling call
0507 08:12:16.406108 (+5005578us) ts_tablet_manager.cc:1120] Deleting tablet 150efae7834740b798af86ae1dcfe445
0507 08:12:16.406117 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.406178 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5005448}
I20240507 08:12:16.407135 6513 tablet_replica.cc:330] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:16.407389 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 150efae7834740b798af86ae1dcfe445 because the tablet was not found. No further retry: Not found: Tablet not found: 150efae7834740b798af86ae1dcfe445
I20240507 08:12:16.407817 6513 raft_consensus.cc:2229] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.408354 6513 raft_consensus.cc:2258] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.408936 6513 ts_tablet_manager.cc:1875] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.394945 5840 tablet_bootstrap.cc:654] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.416035 6513 ts_tablet_manager.cc:1888] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.416185 6513 log.cc:1196] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:12:16.416553 6513 ts_tablet_manager.cc:1909] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.416864 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1022) took 5016 ms. Trace:
I20240507 08:12:16.416993 6513 rpcz_store.cc:276] 0507 08:12:11.400342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.400388 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:16.407092 (+5006704us) ts_tablet_manager.cc:1120] Deleting tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f
0507 08:12:16.407100 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.412649 (+ 5549us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.415964 (+ 3315us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.416847 (+ 883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5006276}
I20240507 08:12:16.417394 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (table default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]) successfully deleted
I20240507 08:12:16.418213 6513 tablet_replica.cc:330] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.418457 6513 raft_consensus.cc:2229] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.418839 6513 raft_consensus.cc:2258] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.419440 6513 ts_tablet_manager.cc:1875] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.420771 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:16.422711 5840 tablet_bootstrap.cc:492] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.422930 5840 ts_tablet_manager.cc:1367] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20240507 08:12:16.427784 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_70c8dabdda144b77a238abeacaa20e6e [id=95b44cbcf26f489c8f4d8e07cc615776]) successfully deleted
I20240507 08:12:16.423672 5840 raft_consensus.cc:348] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.428575 5840 raft_consensus.cc:374] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.429620 5840 raft_consensus.cc:729] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.429929 5840 consensus_queue.cc:260] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.430528 5840 ts_tablet_manager.cc:1398] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240507 08:12:16.396879 7079 rpcz_store.cc:276] 0507 08:12:14.354287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.356678 (+ 2391us) service_pool.cc:225] Handling call
0507 08:12:16.395715 (+2039037us) ts_tablet_manager.cc:1120] Deleting tablet b0542325c80241fca7e317cda64302a9
0507 08:12:16.395727 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.395913 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2038871}
I20240507 08:12:16.433322 5840 tablet_bootstrap.cc:492] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.436405 7079 tablet_replica.cc:330] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:16.436668 7079 raft_consensus.cc:2229] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.436872 7079 raft_consensus.cc:2258] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.438130 7079 ts_tablet_manager.cc:1875] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.438805 5840 tablet_bootstrap.cc:654] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.433040 6513 ts_tablet_manager.cc:1888] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.439085 6513 log.cc:1196] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/6b4c69d38e484f5c819b7dd915bf4410
I20240507 08:12:16.440356 6513 ts_tablet_manager.cc:1909] T 6b4c69d38e484f5c819b7dd915bf4410 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.441282 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1024) took 5040 ms. Trace:
I20240507 08:12:16.395319 5843 ts_tablet_manager.cc:1398] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.392s user 0.115s sys 0.124s
W20240507 08:12:16.396128 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 390731us
I20240507 08:12:16.445876 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 767905763ba14cb8bbc52151ae9f17e6 (table default.Table_4d38bf47908640beb3f6d94ad4446717 [id=84ee77d848974c2d8ccd5d9d518e1ec2]) successfully deleted
I20240507 08:12:16.445940 5843 tablet_bootstrap.cc:492] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:16.446218 5841 tablet_replica.cc:330] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.446427 5841 raft_consensus.cc:2229] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.446774 5841 raft_consensus.cc:2258] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.446996 7378 raft_consensus.cc:482] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.447108 7378 raft_consensus.cc:504] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.447702 7378 leader_election.cc:288] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.447732 5843 tablet_bootstrap.cc:654] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.448264 7378 raft_consensus.cc:482] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.448375 7378 raft_consensus.cc:504] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.448762 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385f4c1eb7f4493a924b813734ddcfdb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:16.448885 7378 leader_election.cc:288] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.449986 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "385f4c1eb7f4493a924b813734ddcfdb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:16.451005 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:16.451398 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242ea35c61404a91b1211c5ad9207728" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:16.452389 5378 leader_election.cc:341] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.453886 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:16.454702 5378 consensus_peers.cc:485] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.454906 5378 leader_election.cc:341] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.455152 5843 tablet_bootstrap.cc:492] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:16.455338 5843 ts_tablet_manager.cc:1367] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
W20240507 08:12:16.455456 5378 consensus_peers.cc:485] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.456203 5843 raft_consensus.cc:348] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.456436 5843 raft_consensus.cc:374] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.456526 5843 raft_consensus.cc:729] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:16.456561 5378 leader_election.cc:341] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.456703 5378 leader_election.cc:302] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.456785 5843 consensus_queue.cc:260] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.448132 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242ea35c61404a91b1211c5ad9207728" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:16.449563 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:16.456270 5656 consensus_peers.cc:485] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b27cc38d0c4c4083900cd6765a5cf6e6. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.457093 5718 raft_consensus.cc:2454] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:16.456938 7378 raft_consensus.cc:2726] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:16.457440 5377 leader_election.cc:341] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:16.457185 5378 leader_election.cc:341] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.457643 5377 leader_election.cc:302] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.457898 7499 raft_consensus.cc:2781] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:16.457975 7499 raft_consensus.cc:482] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:16.458065 7499 raft_consensus.cc:3037] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.458122 5843 ts_tablet_manager.cc:1398] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:16.456992 6513 rpcz_store.cc:276] 0507 08:12:11.400547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.403976 (+ 3429us) service_pool.cc:225] Handling call
0507 08:12:16.418167 (+5014191us) ts_tablet_manager.cc:1120] Deleting tablet 6b4c69d38e484f5c819b7dd915bf4410
0507 08:12:16.418175 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.427617 (+ 9442us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.432954 (+ 5337us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.441261 (+ 8307us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5013572}
I20240507 08:12:16.458451 5843 tablet_bootstrap.cc:492] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:16.458580 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 6b4c69d38e484f5c819b7dd915bf4410 (table default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]) successfully deleted
I20240507 08:12:16.458683 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1026) took 4974 ms. Trace:
I20240507 08:12:16.458757 6513 rpcz_store.cc:276] 0507 08:12:11.484056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.484100 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:16.458569 (+4974469us) ts_tablet_manager.cc:1120] Deleting tablet b6b9232426b948ae8634f7afd65df393
0507 08:12:16.458579 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.458667 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4974290}
I20240507 08:12:16.459941 5843 tablet_bootstrap.cc:654] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:16.460139 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b6b9232426b948ae8634f7afd65df393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6b9232426b948ae8634f7afd65df393 already in progress: creating tablet
I20240507 08:12:16.460239 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1028) took 4956 ms. Trace:
I20240507 08:12:16.460516 6513 rpcz_store.cc:276] 0507 08:12:11.503315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.532866 (+ 29551us) service_pool.cc:225] Handling call
0507 08:12:16.460123 (+4927257us) ts_tablet_manager.cc:1120] Deleting tablet 482e4d0e762b488b8a0cc4e81c2c821d
0507 08:12:16.460132 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.460222 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4926861}
W20240507 08:12:16.461051 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 482e4d0e762b488b8a0cc4e81c2c821d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 482e4d0e762b488b8a0cc4e81c2c821d already in progress: creating tablet
I20240507 08:12:16.461465 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1030) took 4758 ms. Trace:
W20240507 08:12:16.457659 5378 leader_election.cc:341] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.462078 6513 rpcz_store.cc:276] 0507 08:12:11.702891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.702975 (+ 84us) service_pool.cc:225] Handling call
0507 08:12:16.461354 (+4758379us) ts_tablet_manager.cc:1120] Deleting tablet d4fac82e12d0437e950d01dbc4117efa
0507 08:12:16.461365 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.461451 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4758182}
I20240507 08:12:16.462306 5378 leader_election.cc:302] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.464332 5544 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37092 (request call id 182) took 1007 ms. Trace:
I20240507 08:12:16.465963 7378 raft_consensus.cc:2726] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:16.466145 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b0542325c80241fca7e317cda64302a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0542325c80241fca7e317cda64302a9 already in progress: creating tablet
I20240507 08:12:16.466240 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1029) took 4766 ms. Trace:
I20240507 08:12:16.466352 6513 rpcz_store.cc:276] 0507 08:12:11.699814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.707131 (+ 7317us) service_pool.cc:225] Handling call
0507 08:12:16.466144 (+4759013us) ts_tablet_manager.cc:1120] Deleting tablet 46cc8597ca5042ca8e85d99803d07206
0507 08:12:16.466153 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.466225 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4758809}
W20240507 08:12:16.467034 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 46cc8597ca5042ca8e85d99803d07206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cc8597ca5042ca8e85d99803d07206 already in progress: creating tablet
I20240507 08:12:16.467512 6513 tablet_replica.cc:330] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.467746 6513 raft_consensus.cc:2229] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.468087 6513 raft_consensus.cc:2258] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.465149 5544 rpcz_store.cc:276] 0507 08:12:15.456585 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.456645 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:15.499279 (+ 42634us) spinlock_profiling.cc:243] Waited 1.58 ms on lock 0x61800007808c. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2c41a12e 00007f2f2c22079f 00007f2f2ec1e768 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58
0507 08:12:15.511990 (+ 12711us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x6180001d708c. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2c41a12e 00007f2f2c22079f 00007f2f2ec1e768 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58
0507 08:12:16.092212 (+580222us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x6160001a9030. stack: 00007f2f1f6ce756 00007f2f1f6ce510 00007f2f1e70148d 0000000000955804 00000000009557cd 000000000095519a 00007f2f2c220628 00007f2f2ec1e768 00007f2f295aa8e6 00007f2f295aa615 00007f2f263d604e 00007f2f263d4f2e 00007f2f263db405 00007f2f263dbd2f 00007f2f263dbb58 0000000000a60bcc
0507 08:12:16.464075 (+371863us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":251433856}
I20240507 08:12:16.468824 6513 ts_tablet_manager.cc:1875] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.470052 5841 ts_tablet_manager.cc:1875] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.472324 5843 tablet_bootstrap.cc:492] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:16.479157 5843 ts_tablet_manager.cc:1367] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
I20240507 08:12:16.462345 7499 raft_consensus.cc:504] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.480774 7499 leader_election.cc:288] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.481254 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242ea35c61404a91b1211c5ad9207728" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:16.481441 5718 raft_consensus.cc:3037] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.483474 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "242ea35c61404a91b1211c5ad9207728" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:16.484752 5718 raft_consensus.cc:2454] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:12:16.484758 5378 leader_election.cc:341] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.485165 5377 leader_election.cc:302] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.485661 7499 raft_consensus.cc:2781] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:16.485939 7499 raft_consensus.cc:686] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:16.486253 7499 consensus_queue.cc:237] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.488284 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 43644b4405f44bfd9a1afcaedbd68db5
I20240507 08:12:16.488700 5262 catalog_manager.cc:6554] ReplaceTablet: tablet 43644b4405f44bfd9a1afcaedbd68db5 deleted and replaced by tablet 192492075dbf4badb8c70a9546ddd80a
I20240507 08:12:16.489058 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 with delete_type TABLET_DATA_DELETED (replaced by tablet 192492075dbf4badb8c70a9546ddd80a) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.489023 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dac7a1a368f344b9808ba5d8a776b1c9" } new_table_name: "default.Table_4713cfd310954bf59e43b3129f03e8cd" modify_external_catalogs: true
I20240507 08:12:16.489924 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8d8d78277b8044a7b75cc3409a82ef21"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.490391 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d8d78277b8044a7b75cc3409a82ef21 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.491456 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3d79cddfa377454d99a02c32c919b6a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.491904 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c8c312b582bf4a4ab5cd690c377c47f5" } new_table_name: "default.Table_1aa1e107154242ad9d480465921434b1" modify_external_catalogs: true
W20240507 08:12:16.491881 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d79cddfa377454d99a02c32c919b6a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.494381 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82f4891ee00b401cbbedea716a758738 consensus state has no leader
I20240507 08:12:16.495193 5843 raft_consensus.cc:348] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.495757 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet a570e39a52d84d3dbcaeb9ea005d9f41 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:16.496605 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a314de990d004c20810e4ce3cc8c22dd" } modify_external_catalogs: true
I20240507 08:12:16.496662 5843 raft_consensus.cc:374] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.496830 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a314de990d004c20810e4ce3cc8c22dd" } modify_external_catalogs: true
I20240507 08:12:16.496994 5843 raft_consensus.cc:729] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.497440 5843 consensus_queue.cc:260] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.498347 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5ae6b3c712d4df3a94846fb9a4ef837" } new_table_name: "default.Table_2753b56891844d019a4fd40839bc3c59" modify_external_catalogs: true
I20240507 08:12:16.499207 5843 ts_tablet_manager.cc:1398] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240507 08:12:16.500231 5843 tablet_bootstrap.cc:492] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:16.503067 5843 tablet_bootstrap.cc:654] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.507997 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 with delete_type TABLET_DATA_DELETED (replaced by tablet 192492075dbf4badb8c70a9546ddd80a) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:16.516470 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 198a1826a5344117b7c292823c52ec8e consensus state has no leader
I20240507 08:12:16.516409 7079 ts_tablet_manager.cc:1888] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.516865 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_02610397b3da426491da22c6ee52dc68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.517168 7079 log.cc:1196] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/1c7c39ec9fec4331afabb39d1483ef57
W20240507 08:12:16.517369 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02610397b3da426491da22c6ee52dc68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.517990 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ec4d03ca44c54019831f558891aaa6e3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.518432 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec4d03ca44c54019831f558891aaa6e3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.518652 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 140b4601f1a043329c4de612a3ae244c consensus state has no leader
I20240507 08:12:16.521278 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d3a39ab037194d1b9c28f520890fa695"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.521725 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3a39ab037194d1b9c28f520890fa695 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.524505 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f618a8c95a604885884b613b4fdd19c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.524974 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f618a8c95a604885884b613b4fdd19c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.525185 7079 ts_tablet_manager.cc:1909] T 1c7c39ec9fec4331afabb39d1483ef57 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:16.527768 5654 consensus_peers.cc:485] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:16.528309 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1ab1c100102a43598cd1b2cf4f818524 consensus state has no leader
I20240507 08:12:16.528599 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65b762c4d6ac480eb6fbc20827e51a8a" } modify_external_catalogs: true
I20240507 08:12:16.528666 6675 consensus_queue.cc:1035] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:16.528748 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65b762c4d6ac480eb6fbc20827e51a8a" } modify_external_catalogs: true
I20240507 08:12:16.529911 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1091) took 2155 ms. Trace:
I20240507 08:12:16.530351 7079 rpcz_store.cc:276] 0507 08:12:14.374163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.374663 (+ 500us) service_pool.cc:225] Handling call
0507 08:12:16.431834 (+2057171us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:12:16.436346 (+ 4512us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.477119 (+ 40773us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.516024 (+ 38905us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.529891 (+ 13867us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2056935}
W20240507 08:12:16.499353 5378 consensus_peers.cc:485] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.531538 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 1c7c39ec9fec4331afabb39d1483ef57 (table default.Table_32f389693bc94adbaae4903e3d430cc6 [id=8bea5fd171d84b84835562f1fb65659d]) successfully deleted
W20240507 08:12:16.532128 5378 consensus_peers.cc:485] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 364903504aa14d08b050a0cf83b82c71. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.532337 7079 tablet_replica.cc:330] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:16.532977 7079 raft_consensus.cc:2229] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.534051 7079 pending_rounds.cc:62] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Trying to abort 1 pending ops.
I20240507 08:12:16.534269 7079 pending_rounds.cc:68] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924814193897472 op_type: NO_OP noop_request { }
I20240507 08:12:16.534703 7079 raft_consensus.cc:2866] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:16.535027 7079 raft_consensus.cc:2258] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.536525 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f702440833504e4eb59747ee6de0cefb [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 3 got 0
I20240507 08:12:16.537948 7079 ts_tablet_manager.cc:1875] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.538892 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.540030 5420 tablet_service.cc:1467] Processing CreateTablet for tablet c91782a7dabd4484897431c2adbec927 (DEFAULT_TABLE table=default.Table_c63d7d02e1734694abdd64ac83426682 [id=e464111e2fef4a6bb2747f5290c8c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.540506 5685 tablet_service.cc:1467] Processing CreateTablet for tablet c91782a7dabd4484897431c2adbec927 (DEFAULT_TABLE table=default.Table_c63d7d02e1734694abdd64ac83426682 [id=e464111e2fef4a6bb2747f5290c8c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.540887 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91782a7dabd4484897431c2adbec927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.541031 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91782a7dabd4484897431c2adbec927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.541235 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.541909 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 26b3814b94514b2483e33bdffa5e3242 (DEFAULT_TABLE table=default.Table_7fa1f6e175bf4f0d9146ed3ca065146a [id=a33420282a9b4a00bcd8ad59c332e448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.542209 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b3814b94514b2483e33bdffa5e3242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.542622 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 310c601abead42e0bd2a2df1a0e16ec2
I20240507 08:12:16.546142 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.546293 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 26b3814b94514b2483e33bdffa5e3242 (DEFAULT_TABLE table=default.Table_7fa1f6e175bf4f0d9146ed3ca065146a [id=a33420282a9b4a00bcd8ad59c332e448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.546653 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b3814b94514b2483e33bdffa5e3242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.547530 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.547705 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 with delete_type TABLET_DATA_DELETED (replaced by tablet 8f9d270dcfe84128a31fe48d5f2be090) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.547861 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.548729 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.553444 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 08aac6e88b7c41d8a59d1e12e7f6c14e
I20240507 08:12:16.553480 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2af8358d14456fbc7755fa9cfde05a (DEFAULT_TABLE table=default.Table_530de072d6794063a4670b0748d8f4f0 [id=7152c1a7ae194911a686079be7620865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.553697 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.553915 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2af8358d14456fbc7755fa9cfde05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.555769 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2af8358d14456fbc7755fa9cfde05a (DEFAULT_TABLE table=default.Table_530de072d6794063a4670b0748d8f4f0 [id=7152c1a7ae194911a686079be7620865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.556169 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2af8358d14456fbc7755fa9cfde05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.560351 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 5e33890b1ec9432f9b29694b71d8e9b4 (DEFAULT_TABLE table=default.Table_43d29dc135d54b5bb43129f3ee487d07 [id=691c615801cf4e1eac0c9c3546a49045]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.560732 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e33890b1ec9432f9b29694b71d8e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.561864 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef4375925474e789a39a2c28fa6adfe (DEFAULT_TABLE table=default.Table_378c906ca8ff4cf199a515389bd939d8 [id=597d8c34cbe24dc7b90fcf7f3fdf6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.562501 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 6a2af8358d14456fbc7755fa9cfde05a (DEFAULT_TABLE table=default.Table_530de072d6794063a4670b0748d8f4f0 [id=7152c1a7ae194911a686079be7620865]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.562682 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 26b3814b94514b2483e33bdffa5e3242 (DEFAULT_TABLE table=default.Table_7fa1f6e175bf4f0d9146ed3ca065146a [id=a33420282a9b4a00bcd8ad59c332e448]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.562824 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a2af8358d14456fbc7755fa9cfde05a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.563002 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 310c601abead42e0bd2a2df1a0e16ec2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.563180 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.563231 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef4375925474e789a39a2c28fa6adfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.563736 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26b3814b94514b2483e33bdffa5e3242. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.564493 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 5e33890b1ec9432f9b29694b71d8e9b4 (DEFAULT_TABLE table=default.Table_43d29dc135d54b5bb43129f3ee487d07 [id=691c615801cf4e1eac0c9c3546a49045]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.564846 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e33890b1ec9432f9b29694b71d8e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.565444 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.570331 7100 consensus_queue.cc:1035] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:16.571635 7378 raft_consensus.cc:482] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.571758 7378 raft_consensus.cc:504] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.572315 7378 leader_election.cc:288] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.574446 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 5e33890b1ec9432f9b29694b71d8e9b4 (DEFAULT_TABLE table=default.Table_43d29dc135d54b5bb43129f3ee487d07 [id=691c615801cf4e1eac0c9c3546a49045]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.574805 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e33890b1ec9432f9b29694b71d8e9b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.577286 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:16.577744 5377 leader_election.cc:341] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.578963 6513 ts_tablet_manager.cc:1888] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.579072 6513 log.cc:1196] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/31251576beee4a8a81a453b721c9c058
I20240507 08:12:16.579420 6513 ts_tablet_manager.cc:1909] T 31251576beee4a8a81a453b721c9c058 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.579926 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 08aac6e88b7c41d8a59d1e12e7f6c14e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.581634 5840 tablet_bootstrap.cc:492] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.583847 5840 ts_tablet_manager.cc:1367] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.151s user 0.014s sys 0.010s
I20240507 08:12:16.585605 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1032) took 4781 ms. Trace:
I20240507 08:12:16.585688 6513 rpcz_store.cc:276] 0507 08:12:11.804162 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.804533 (+ 371us) service_pool.cc:225] Handling call
0507 08:12:16.467468 (+4662935us) ts_tablet_manager.cc:1120] Deleting tablet 31251576beee4a8a81a453b721c9c058
0507 08:12:16.467477 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.559347 (+ 91870us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.578887 (+ 19540us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.585560 (+ 6673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4659725}
I20240507 08:12:16.586177 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 31251576beee4a8a81a453b721c9c058 (table default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]) successfully deleted
I20240507 08:12:16.586992 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1034) took 4710 ms. Trace:
I20240507 08:12:16.587500 6513 rpcz_store.cc:276] 0507 08:12:11.876999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.877066 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:16.586856 (+4709790us) ts_tablet_manager.cc:1120] Deleting tablet e3783fc7fe434696898ba14ab77a9058
0507 08:12:16.586866 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.586976 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4708650}
I20240507 08:12:16.587855 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1033) took 4711 ms. Trace:
I20240507 08:12:16.587776 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d2db26a18c77448aba96e29400de518e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.587942 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet e3783fc7fe434696898ba14ab77a9058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3783fc7fe434696898ba14ab77a9058 already in progress: creating tablet
I20240507 08:12:16.587941 6513 rpcz_store.cc:276] 0507 08:12:11.876844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.877159 (+ 315us) service_pool.cc:225] Handling call
0507 08:12:16.587748 (+4710589us) ts_tablet_manager.cc:1120] Deleting tablet aafd1dc2eaf84390b072f01a9e76232d
0507 08:12:16.587758 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.587840 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4710405}
I20240507 08:12:16.588209 6513 tablet_replica.cc:330] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:16.588328 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2db26a18c77448aba96e29400de518e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.588411 6513 raft_consensus.cc:2229] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:16.588574 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet aafd1dc2eaf84390b072f01a9e76232d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aafd1dc2eaf84390b072f01a9e76232d already in progress: creating tablet
I20240507 08:12:16.588706 6513 pending_rounds.cc:62] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:16.588757 6513 pending_rounds.cc:68] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924809573199872 op_type: NO_OP noop_request { }
I20240507 08:12:16.588873 6513 raft_consensus.cc:2866] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:16.588971 6513 raft_consensus.cc:2258] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.588900 5840 raft_consensus.cc:348] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.589180 5840 raft_consensus.cc:374] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.589274 5840 raft_consensus.cc:729] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.590925 5840 consensus_queue.cc:260] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.594118 5840 ts_tablet_manager.cc:1398] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20240507 08:12:16.594172 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:16.594851 5378 leader_election.cc:341] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.594982 5378 leader_election.cc:302] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.595194 7499 raft_consensus.cc:2726] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.595458 6513 ts_tablet_manager.cc:1875] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.595616 5840 tablet_bootstrap.cc:492] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.596851 5840 tablet_bootstrap.cc:654] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:16.598654 5654 consensus_peers.cc:485] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.600903 7079 ts_tablet_manager.cc:1888] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.601516 7079 log.cc:1196] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/79524df2415146ab926425c42068626c
I20240507 08:12:16.603081 7079 ts_tablet_manager.cc:1909] T 79524df2415146ab926425c42068626c P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:16.606226 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1092) took 2231 ms. Trace:
I20240507 08:12:16.606572 7079 rpcz_store.cc:276] 0507 08:12:14.374250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.376520 (+ 2270us) service_pool.cc:225] Handling call
0507 08:12:16.532093 (+2155573us) ts_tablet_manager.cc:1120] Deleting tablet 79524df2415146ab926425c42068626c
0507 08:12:16.532193 (+ 100us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.582415 (+ 50222us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.600503 (+ 18088us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.606114 (+ 5611us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155371}
I20240507 08:12:16.607890 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 79524df2415146ab926425c42068626c (table default.Table_16b866688308450cb4d4f81ccafa1afb [id=35093b3de3164b9c809aae8b5da3db54]) successfully deleted
I20240507 08:12:16.608829 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 with delete_type TABLET_DATA_DELETED (replaced by tablet 192492075dbf4badb8c70a9546ddd80a) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.608645 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1093) took 2225 ms. Trace:
I20240507 08:12:16.609248 7079 rpcz_store.cc:276] 0507 08:12:14.383558 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.386822 (+ 3264us) service_pool.cc:225] Handling call
0507 08:12:16.608136 (+2221314us) ts_tablet_manager.cc:1120] Deleting tablet ab350a3f4f3341c2ad0dda3ce054e199
0507 08:12:16.608247 (+ 111us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.608540 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2220990}
W20240507 08:12:16.610155 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ab350a3f4f3341c2ad0dda3ce054e199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab350a3f4f3341c2ad0dda3ce054e199 already in progress: creating tablet
I20240507 08:12:16.611225 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1094) took 2165 ms. Trace:
I20240507 08:12:16.611552 7079 rpcz_store.cc:276] 0507 08:12:14.445205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.447232 (+ 2027us) service_pool.cc:225] Handling call
0507 08:12:16.610637 (+2163405us) ts_tablet_manager.cc:1120] Deleting tablet 55bacb9fd24b46709704eb852e051eb1
0507 08:12:16.610646 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.610914 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2163211}
W20240507 08:12:16.612394 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 55bacb9fd24b46709704eb852e051eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55bacb9fd24b46709704eb852e051eb1 already in progress: creating tablet
I20240507 08:12:16.612923 7079 tablet_replica.cc:330] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:16.613628 7079 raft_consensus.cc:2229] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:16.614389 5656 consensus_peers.cc:485] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 831153d2b89e43f8b08b086f7a8f3a54. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.614449 7079 raft_consensus.cc:2258] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.617691 7079 ts_tablet_manager.cc:1875] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.622516 5554 tablet_service.cc:1467] Processing CreateTablet for tablet c91782a7dabd4484897431c2adbec927 (DEFAULT_TABLE table=default.Table_c63d7d02e1734694abdd64ac83426682 [id=e464111e2fef4a6bb2747f5290c8c252]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.622898 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c91782a7dabd4484897431c2adbec927. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.597405 7529 tablet.cc:1722] T 645ae26bb26f43d298afe98817d88fa4 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:16.626369 7499 consensus_queue.cc:1035] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:16.628742 5839 tablet_bootstrap.cc:654] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.633460 7101 raft_consensus.cc:482] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.633720 7101 raft_consensus.cc:504] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.638096 7101 leader_election.cc:288] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.638464 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:16.638525 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:16.638710 5444 raft_consensus.cc:2454] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:16.639075 5656 leader_election.cc:302] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:16.639212 5654 leader_election.cc:341] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.639773 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 33b62a463d8f416f8407dd866c5c993b (table default.Table_840fddb88c744ccc927287f323d16cc8 [id=bb7a82da939941f6bdd819b36218f7df]). Expected version 1 got 0
I20240507 08:12:16.640307 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:16.640655 5258 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (table default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3]). Expected version 2 got 0
I20240507 08:12:16.640784 7079 ts_tablet_manager.cc:1888] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:16.641333 7079 log.cc:1196] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/4306751d16444ec5948c2ba5831063f4
I20240507 08:12:16.642241 7100 raft_consensus.cc:2781] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:16.642328 7100 raft_consensus.cc:482] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:16.642405 7100 raft_consensus.cc:3037] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.643035 7079 ts_tablet_manager.cc:1909] T 4306751d16444ec5948c2ba5831063f4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:16.645888 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1095) took 1940 ms. Trace:
I20240507 08:12:16.645946 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab94b1a09dd4969b928b017229ce4bf with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:09 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.646293 7079 rpcz_store.cc:276] 0507 08:12:14.705707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.705765 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:16.612783 (+1907018us) ts_tablet_manager.cc:1120] Deleting tablet 4306751d16444ec5948c2ba5831063f4
0507 08:12:16.612883 (+ 100us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.628212 (+ 15329us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.640387 (+ 12175us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.645863 (+ 5476us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1906817}
I20240507 08:12:16.647434 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]) successfully deleted
I20240507 08:12:16.647387 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.648429 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1096) took 1796 ms. Trace:
I20240507 08:12:16.648743 7079 rpcz_store.cc:276] 0507 08:12:14.852244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.852299 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:16.648132 (+1795833us) ts_tablet_manager.cc:1120] Deleting tablet 093fb3986a674b93a435408c7ec85500
0507 08:12:16.648145 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.648414 (+ 269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1795536}
W20240507 08:12:16.649461 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 093fb3986a674b93a435408c7ec85500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 093fb3986a674b93a435408c7ec85500 already in progress: creating tablet
I20240507 08:12:16.651377 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef4375925474e789a39a2c28fa6adfe (DEFAULT_TABLE table=default.Table_378c906ca8ff4cf199a515389bd939d8 [id=597d8c34cbe24dc7b90fcf7f3fdf6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.651852 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef4375925474e789a39a2c28fa6adfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.652211 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 73f82413951c48a89e7db8fe9e50c749
I20240507 08:12:16.652876 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9669c1169b5244218383f7aa7fbc3b85
I20240507 08:12:16.653710 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f82413951c48a89e7db8fe9e50c749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.653867 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.653867 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f82413951c48a89e7db8fe9e50c749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.654780 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 994f50ea19a2445288663b7d7b7ad291
I20240507 08:12:16.655658 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 994f50ea19a2445288663b7d7b7ad291 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.656651 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 994f50ea19a2445288663b7d7b7ad291 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.657214 7531 tablet.cc:1722] T 645ae26bb26f43d298afe98817d88fa4 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:16.657749 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1097) took 1750 ms. Trace:
I20240507 08:12:16.657836 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.658166 7079 rpcz_store.cc:276] 0507 08:12:14.907304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.907364 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:16.649977 (+1742613us) ts_tablet_manager.cc:1120] Deleting tablet 19b77b6ef778454e91105e1e9b60020d
0507 08:12:16.657031 (+ 7054us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.657543 (+ 512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742426}
W20240507 08:12:16.659049 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 19b77b6ef778454e91105e1e9b60020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b77b6ef778454e91105e1e9b60020d already in progress: creating tablet
I20240507 08:12:16.659806 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1099) took 1621 ms. Trace:
I20240507 08:12:16.660254 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 73f82413951c48a89e7db8fe9e50c749 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.660159 7079 rpcz_store.cc:276] 0507 08:12:15.038018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.038066 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:16.659437 (+1621371us) ts_tablet_manager.cc:1120] Deleting tablet 62d6eac6baa249eb847c5625d64f998f
0507 08:12:16.659446 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.659701 (+ 255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621186}
W20240507 08:12:16.661406 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 62d6eac6baa249eb847c5625d64f998f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d6eac6baa249eb847c5625d64f998f already in progress: creating tablet
I20240507 08:12:16.661989 7539 tablet.cc:1722] T 33b62a463d8f416f8407dd866c5c993b P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:16.660513 7529 tablet.cc:1722] T 33b62a463d8f416f8407dd866c5c993b P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:16.662334 7538 tablet.cc:1722] T 33b62a463d8f416f8407dd866c5c993b P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:16.665329 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac127bae0954e57b702d7f257eda695 (DEFAULT_TABLE table=default.Table_e3dfd4eec9cf49a3a875332a653befd8 [id=cbdb52bba6cc4d9b840b2e955087622e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.665709 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac127bae0954e57b702d7f257eda695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.667906 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 9669c1169b5244218383f7aa7fbc3b85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.669849 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 3ef4375925474e789a39a2c28fa6adfe (DEFAULT_TABLE table=default.Table_378c906ca8ff4cf199a515389bd939d8 [id=597d8c34cbe24dc7b90fcf7f3fdf6611]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.670208 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ef4375925474e789a39a2c28fa6adfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.670943 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 994f50ea19a2445288663b7d7b7ad291 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.678566 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 17c6f752a0d4485e9606a5570a0d6c96 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:16.681074 5249 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 82f4891ee00b401cbbedea716a758738 (table default.Table_7568e0a7af0845dc863059434b0ed0c1 [id=c8cf84c4109a46b0a6ba90228486e423], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 82f4891ee00b401cbbedea716a758738 consensus state has no leader
I20240507 08:12:16.683677 7100 raft_consensus.cc:504] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.684317 7100 leader_election.cc:288] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.685351 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:16.685799 5443 raft_consensus.cc:3037] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.688917 5443 raft_consensus.cc:2454] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:16.689193 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eb9840182697421db11214ad83ad8b24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.689762 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb9840182697421db11214ad83ad8b24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.691637 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b0948279632f4c32a83404acd89ae917" } new_table_name: "default.Table_b910222ad6bb422d9827b291b4ae80ac" modify_external_catalogs: true
I20240507 08:12:16.694356 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3004daed58924ac899ae89618d420197"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.694941 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3004daed58924ac899ae89618d420197 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.662663 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1103) took 1602 ms. Trace:
I20240507 08:12:16.696367 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_77f1b8f4de134bf8acda3315c2f08293" } new_table_name: "default.Table_5b20d4f1f4a34a9385701d8a6639e647" modify_external_catalogs: true
I20240507 08:12:16.696302 7079 rpcz_store.cc:276] 0507 08:12:15.060504 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.065662 (+ 5158us) service_pool.cc:225] Handling call
0507 08:12:16.662072 (+1596410us) ts_tablet_manager.cc:1120] Deleting tablet a24813c72abd41c3830c7392c6149151
0507 08:12:16.662247 (+ 175us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.662647 (+ 400us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1596183}
I20240507 08:12:16.665349 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac127bae0954e57b702d7f257eda695 (DEFAULT_TABLE table=default.Table_e3dfd4eec9cf49a3a875332a653befd8 [id=cbdb52bba6cc4d9b840b2e955087622e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.685020 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99e4514039304f0696c69a057db0ea6e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:16.691118 5561 tablet_service.cc:1467] Processing CreateTablet for tablet d1f0d52867fe48dba332c71ad4cef40b (DEFAULT_TABLE table=default.Table_74382721d3fd4f3dac0a931308d156a4 [id=479627df82c241a4acb5ef9b523fba73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.691190 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 582714ecd2404c39b2d51f7b1009b2f3 (DEFAULT_TABLE table=default.Table_5844bcf059ea43c6a4e0c092a9f8d3d5 [id=d61e41fe6d1546ebb29b252a09298789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.696841 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1104) took 1622 ms. Trace:
W20240507 08:12:16.696908 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a24813c72abd41c3830c7392c6149151 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24813c72abd41c3830c7392c6149151 already in progress: creating tablet
I20240507 08:12:16.696929 7079 rpcz_store.cc:276] 0507 08:12:15.074647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.074705 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:16.696598 (+1621893us) ts_tablet_manager.cc:1120] Deleting tablet db92da02e5af49f5a13a814f3ce4f9c3
0507 08:12:16.696706 (+ 108us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.696825 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17792,"tablet-delete.queue_time_us":1621704}
W20240507 08:12:16.697320 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet db92da02e5af49f5a13a814f3ce4f9c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db92da02e5af49f5a13a814f3ce4f9c3 already in progress: creating tablet
I20240507 08:12:16.697423 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac127bae0954e57b702d7f257eda695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.698040 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3c8c77f7212e495eb588a34553d6f7d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.698446 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c8c77f7212e495eb588a34553d6f7d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.700167 6513 ts_tablet_manager.cc:1888] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.700300 6513 log.cc:1196] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:12:16.700664 6513 ts_tablet_manager.cc:1909] T a56cb45381604ac9a69e6f3edaa4eefc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.701915 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f0d52867fe48dba332c71ad4cef40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.705972 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1117) took 1504 ms. Trace:
I20240507 08:12:16.706079 7079 rpcz_store.cc:276] 0507 08:12:15.201224 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.202779 (+ 1555us) service_pool.cc:225] Handling call
0507 08:12:16.697169 (+1494390us) ts_tablet_manager.cc:1120] Deleting tablet cc798eb4e02c456880ef3014ebbb7e24
0507 08:12:16.705807 (+ 8638us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.705949 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494208}
I20240507 08:12:16.706477 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1118) took 1476 ms. Trace:
I20240507 08:12:16.706552 7079 rpcz_store.cc:276] 0507 08:12:15.229676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.231815 (+ 2139us) service_pool.cc:225] Handling call
0507 08:12:16.706374 (+1474559us) ts_tablet_manager.cc:1120] Deleting tablet f748727d3a88442d90cf81bae48acd36
0507 08:12:16.706384 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.706462 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1474393}
I20240507 08:12:16.706815 7079 tablet_replica.cc:330] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:16.707082 7079 raft_consensus.cc:2229] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.707243 7079 raft_consensus.cc:2258] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.707847 7079 ts_tablet_manager.cc:1875] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.710088 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582714ecd2404c39b2d51f7b1009b2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.715677 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 582714ecd2404c39b2d51f7b1009b2f3 (DEFAULT_TABLE table=default.Table_5844bcf059ea43c6a4e0c092a9f8d3d5 [id=d61e41fe6d1546ebb29b252a09298789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.716010 5839 tablet_bootstrap.cc:492] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:16.716061 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582714ecd2404c39b2d51f7b1009b2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.716300 5839 ts_tablet_manager.cc:1367] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.651s user 0.013s sys 0.001s
I20240507 08:12:16.716790 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddcd0d1f884aee9a66a52573eb6011 (DEFAULT_TABLE table=default.Table_4d223c1ac6e944a4bff2404a133840dc [id=fb4c4b880ff947d3be1ba4311a96be86]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:16.717693 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cc798eb4e02c456880ef3014ebbb7e24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc798eb4e02c456880ef3014ebbb7e24 already in progress: creating tablet
I20240507 08:12:16.717978 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e9828e7d20ab4c959f6963420d4fa85e" } new_table_name: "default.Table_12a6149f27754dcfa4b02b2a1a53b6e5" modify_external_catalogs: true
I20240507 08:12:16.717618 5839 raft_consensus.cc:348] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.719885 5839 raft_consensus.cc:374] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.720028 5698 tablet_service.cc:1467] Processing CreateTablet for tablet d1f0d52867fe48dba332c71ad4cef40b (DEFAULT_TABLE table=default.Table_74382721d3fd4f3dac0a931308d156a4 [id=479627df82c241a4acb5ef9b523fba73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.720364 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f0d52867fe48dba332c71ad4cef40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.719993 5839 raft_consensus.cc:729] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.720587 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_47188cab4dee4904b2a5099a06b2a652" } new_table_name: "default.Table_954d2355c091469cb6dbc10641b187a6" modify_external_catalogs: true
I20240507 08:12:16.720784 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7ab1a575a68349d4a23fb56a2fd5098b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.720939 5839 consensus_queue.cc:260] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:16.721241 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab1a575a68349d4a23fb56a2fd5098b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.730090 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 17c6f752a0d4485e9606a5570a0d6c96
I20240507 08:12:16.730536 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 17c6f752a0d4485e9606a5570a0d6c96 deleted and replaced by tablet 94eafbab0c304f24abe35d53dcfd6919
W20240507 08:12:16.731658 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb5a077be5574cb8bc7a2347c6d3af00 consensus state has no leader
I20240507 08:12:16.732175 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_108b5f297f7f42a0a09409c1a3e4b975"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.732643 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_108b5f297f7f42a0a09409c1a3e4b975 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.694499 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddcd0d1f884aee9a66a52573eb6011 (DEFAULT_TABLE table=default.Table_4d223c1ac6e944a4bff2404a133840dc [id=fb4c4b880ff947d3be1ba4311a96be86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.733229 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddcd0d1f884aee9a66a52573eb6011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.734519 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 8ac127bae0954e57b702d7f257eda695 (DEFAULT_TABLE table=default.Table_e3dfd4eec9cf49a3a875332a653befd8 [id=cbdb52bba6cc4d9b840b2e955087622e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.734898 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8ac127bae0954e57b702d7f257eda695. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.736706 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c35d3c43d02b421186d2a6a8548ac31f" } new_table_name: "default.Table_9d3174b2f05d48bcae3a5624a768b09a" modify_external_catalogs: true
I20240507 08:12:16.737720 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddcd0d1f884aee9a66a52573eb6011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.738397 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:16.739027 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (table default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3]). Expected version 2 got 0
I20240507 08:12:16.740474 5839 ts_tablet_manager.cc:1398] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240507 08:12:16.663460 7418 raft_consensus.cc:482] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.741559 7418 raft_consensus.cc:504] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.742264 7418 leader_election.cc:288] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:16.741034 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94eafbab0c304f24abe35d53dcfd6919) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.741164 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 31251576beee4a8a81a453b721c9c058 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.694622 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 982fb83171db41ea9d1572d68e830778 (DEFAULT_TABLE table=default.Table_8f670d8a09204a8fb9319c3d39bd175f [id=900e33cbb529457dbebfb81e176aec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.743599 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982fb83171db41ea9d1572d68e830778. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.746599 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 124ea88554ec4182a6682457cd58dfe1 consensus state has no leader
I20240507 08:12:16.742992 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
W20240507 08:12:16.748366 5517 leader_election.cc:341] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: b2df872d04b94c30872b03367df3cc31
I20240507 08:12:16.749506 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c13e53ccc0144cb4a2fd3cce833674f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.749910 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bd9f3573c06a41fcbf0fd31c87414464"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.750336 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd9f3573c06a41fcbf0fd31c87414464 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.749770 5839 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:16.751343 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_881768862fbe460d961c048ed94bd724"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.751770 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_881768862fbe460d961c048ed94bd724 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.751767 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:16.752753 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_615d90859dd74c9aa813e233b63dd68e" } modify_external_catalogs: true
I20240507 08:12:16.753369 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_615d90859dd74c9aa813e233b63dd68e" } modify_external_catalogs: true
I20240507 08:12:16.758805 7079 ts_tablet_manager.cc:1888] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:16.758937 7079 log.cc:1196] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/d451d03349794726a71c255a7f177436
I20240507 08:12:16.759311 7079 ts_tablet_manager.cc:1909] T d451d03349794726a71c255a7f177436 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:16.759328 5654 leader_election.cc:341] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.760344 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94eafbab0c304f24abe35d53dcfd6919) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:16.750025 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c13e53ccc0144cb4a2fd3cce833674f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.765537 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2572985bc2444dc7b051f8fe008e3ad4 consensus state has no leader
I20240507 08:12:16.767707 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9bdc371ba30a4c879ec337f2a1d3e90e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.768177 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bdc371ba30a4c879ec337f2a1d3e90e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.769333 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d718aa8fe99041688964d1a05cef49a0 consensus state has no leader
I20240507 08:12:16.769654 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e53530be7694370bbab56e5fdc054c0" } modify_external_catalogs: true
I20240507 08:12:16.769816 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2e53530be7694370bbab56e5fdc054c0" } modify_external_catalogs: true
W20240507 08:12:16.771922 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe8e4d53d3b44387b5ca12d5473947be consensus state has no leader
W20240507 08:12:16.778985 5520 leader_election.cc:341] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b2df872d04b94c30872b03367df3cc31
I20240507 08:12:16.779105 5520 leader_election.cc:302] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:16.779465 7418 raft_consensus.cc:2726] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.780023 5839 tablet_bootstrap.cc:654] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.781469 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet bee9e434083f4fa6859cfe2b7cca16e5
I20240507 08:12:16.782269 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.782786 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1035) took 4902 ms. Trace:
I20240507 08:12:16.782866 6513 rpcz_store.cc:276] 0507 08:12:11.880268 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.880316 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:16.588169 (+4707853us) ts_tablet_manager.cc:1120] Deleting tablet a56cb45381604ac9a69e6f3edaa4eefc
0507 08:12:16.588180 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.609686 (+ 21506us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.700061 (+ 90375us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.782770 (+ 82709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4707652}
I20240507 08:12:16.783893 5840 tablet_bootstrap.cc:492] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.783955 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.785470 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet a56cb45381604ac9a69e6f3edaa4eefc (table default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]) successfully deleted
I20240507 08:12:16.787698 5840 ts_tablet_manager.cc:1367] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.192s user 0.006s sys 0.004s
I20240507 08:12:16.788484 5840 raft_consensus.cc:348] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.794301 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1126) took 1463 ms. Trace:
I20240507 08:12:16.794450 7079 rpcz_store.cc:276] 0507 08:12:15.331167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.331216 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:16.706767 (+1375551us) ts_tablet_manager.cc:1120] Deleting tablet d451d03349794726a71c255a7f177436
0507 08:12:16.706776 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.728582 (+ 21806us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.758700 (+ 30118us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.794281 (+ 35581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1375341}
I20240507 08:12:16.800223 5840 raft_consensus.cc:374] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.800521 5840 raft_consensus.cc:729] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.800910 5840 consensus_queue.cc:260] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.801929 5840 ts_tablet_manager.cc:1398] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20240507 08:12:16.804020 5840 tablet_bootstrap.cc:492] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.787458 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1037) took 4893 ms. Trace:
I20240507 08:12:16.804895 6513 rpcz_store.cc:276] 0507 08:12:11.893865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.893930 (+ 65us) service_pool.cc:225] Handling call
0507 08:12:16.787225 (+4893295us) ts_tablet_manager.cc:1120] Deleting tablet 323b96ae7d1e471998a39f3be0857ce7
0507 08:12:16.787323 (+ 98us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.787441 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4893051}
W20240507 08:12:16.805383 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 323b96ae7d1e471998a39f3be0857ce7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323b96ae7d1e471998a39f3be0857ce7 already in progress: creating tablet
I20240507 08:12:16.805414 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1036) took 4911 ms. Trace:
I20240507 08:12:16.805493 6513 rpcz_store.cc:276] 0507 08:12:11.893745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.894150 (+ 405us) service_pool.cc:225] Handling call
0507 08:12:16.805273 (+4911123us) ts_tablet_manager.cc:1120] Deleting tablet cacfe980b2aa4a60ad05bbe102a94266
0507 08:12:16.805283 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.805396 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4910912}
I20240507 08:12:16.805552 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet d451d03349794726a71c255a7f177436 (table default.Table_722dbcaa48584561970775496bbde987 [id=8009981986dd4572bbc47ff46f469093]) successfully deleted
I20240507 08:12:16.805965 7079 tablet_replica.cc:330] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:16.805984 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet cacfe980b2aa4a60ad05bbe102a94266 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cacfe980b2aa4a60ad05bbe102a94266 already in progress: creating tablet
I20240507 08:12:16.806306 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1038) took 4904 ms. Trace:
I20240507 08:12:16.806327 7079 raft_consensus.cc:2229] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.806403 6513 rpcz_store.cc:276] 0507 08:12:11.902243 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.903055 (+ 812us) service_pool.cc:225] Handling call
0507 08:12:16.806205 (+4903150us) ts_tablet_manager.cc:1120] Deleting tablet 6ad4ed7878454fea930b3982bcb6eed4
0507 08:12:16.806215 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.806290 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4902876}
I20240507 08:12:16.806656 7079 raft_consensus.cc:2258] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:16.806775 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 6ad4ed7878454fea930b3982bcb6eed4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ad4ed7878454fea930b3982bcb6eed4 already in progress: creating tablet
I20240507 08:12:16.807135 6513 tablet_replica.cc:330] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.807317 6513 raft_consensus.cc:2229] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.807745 6513 raft_consensus.cc:2258] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.808300 6513 ts_tablet_manager.cc:1875] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.809728 5840 tablet_bootstrap.cc:654] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.811712 5839 tablet_bootstrap.cc:492] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:16.811841 5839 ts_tablet_manager.cc:1367] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20240507 08:12:16.813117 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 982fb83171db41ea9d1572d68e830778 (DEFAULT_TABLE table=default.Table_8f670d8a09204a8fb9319c3d39bd175f [id=900e33cbb529457dbebfb81e176aec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.813484 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982fb83171db41ea9d1572d68e830778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.813333 5839 raft_consensus.cc:348] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.814366 5839 raft_consensus.cc:374] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.814908 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 14944e46cb1c4460b1e2f154a8e8a085 (DEFAULT_TABLE table=default.Table_5e6b208d88324e1d913e9c07d26d71e4 [id=975d5db97b8c452692a4ff484b1e9687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.815186 5839 raft_consensus.cc:729] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.815598 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14944e46cb1c4460b1e2f154a8e8a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.814689 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f (DEFAULT_TABLE table=default.Table_54152e52f0df4b3281c833102a7d9aff [id=55e6acafaa384f6795e99cdbcf0c8731]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.815795 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a68badb3b5644ed8b007b2b04bc4f6e1
I20240507 08:12:16.816231 5561 tablet_service.cc:1467] Processing CreateTablet for tablet fe8e4d53d3b44387b5ca12d5473947be (DEFAULT_TABLE table=default.Table_9d3174b2f05d48bcae3a5624a768b09a [id=e30f4b8dd1004b6cb23c5ec60f9f249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.816315 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.816677 5259 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_840fddb88c744ccc927287f323d16cc8 [id=bb7a82da939941f6bdd819b36218f7df] alter complete (version 1)
I20240507 08:12:16.816872 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.817184 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8e4d53d3b44387b5ca12d5473947be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.815497 5839 consensus_queue.cc:260] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.818410 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.818316 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f (DEFAULT_TABLE table=default.Table_54152e52f0df4b3281c833102a7d9aff [id=55e6acafaa384f6795e99cdbcf0c8731]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.818807 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.828280 5839 ts_tablet_manager.cc:1398] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240507 08:12:16.830117 5839 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:16.831758 5839 tablet_bootstrap.cc:654] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.844803 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:16.848094 7079 ts_tablet_manager.cc:1875] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.694444 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 2cddcd0d1f884aee9a66a52573eb6011 (DEFAULT_TABLE table=default.Table_4d223c1ac6e944a4bff2404a133840dc [id=fb4c4b880ff947d3be1ba4311a96be86]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.695014 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 582714ecd2404c39b2d51f7b1009b2f3 (DEFAULT_TABLE table=default.Table_5844bcf059ea43c6a4e0c092a9f8d3d5 [id=d61e41fe6d1546ebb29b252a09298789]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.853771 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 582714ecd2404c39b2d51f7b1009b2f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.853796 5656 leader_election.cc:302] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.854193 7100 raft_consensus.cc:2781] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:16.854310 7100 raft_consensus.cc:686] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:16.854521 7100 consensus_queue.cc:237] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.704433 5424 tablet_service.cc:1467] Processing CreateTablet for tablet d1f0d52867fe48dba332c71ad4cef40b (DEFAULT_TABLE table=default.Table_74382721d3fd4f3dac0a931308d156a4 [id=479627df82c241a4acb5ef9b523fba73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.709893 5841 ts_tablet_manager.cc:1888] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.733047 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 17c6f752a0d4485e9606a5570a0d6c96 with delete_type TABLET_DATA_DELETED (replaced by tablet 94eafbab0c304f24abe35d53dcfd6919) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.782407 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet bee9e434083f4fa6859cfe2b7cca16e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.857707 5841 log.cc:1196] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:12:16.786265 7497 raft_consensus.cc:482] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.857844 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1f0d52867fe48dba332c71ad4cef40b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.857865 7497 raft_consensus.cc:504] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.858130 5841 ts_tablet_manager.cc:1909] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.858417 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1079) took 3363 ms. Trace:
I20240507 08:12:16.858480 5841 rpcz_store.cc:276] 0507 08:12:13.495141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.501290 (+ 6149us) service_pool.cc:225] Handling call
0507 08:12:16.446166 (+2944876us) ts_tablet_manager.cc:1120] Deleting tablet 84212d6cb2c742c1a0f3fdae8be8ebe1
0507 08:12:16.446177 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.473031 (+ 26854us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.709757 (+236726us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.858399 (+148642us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2944722}
I20240507 08:12:16.858583 7497 leader_election.cc:288] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.858889 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1093) took 3257 ms. Trace:
I20240507 08:12:16.858953 5841 rpcz_store.cc:276] 0507 08:12:13.601865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.601918 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:16.858761 (+3256843us) ts_tablet_manager.cc:1120] Deleting tablet 497ab12edc724eef8d6e7b631662a4ab
0507 08:12:16.858769 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.858876 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3255507}
I20240507 08:12:16.859160 5841 tablet_replica.cc:330] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.859319 5841 raft_consensus.cc:2229] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.859453 5841 raft_consensus.cc:2258] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.859390 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534708f1abd4a33965eea92bdc0d0de" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:16.859728 5840 tablet_bootstrap.cc:492] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.859848 5840 ts_tablet_manager.cc:1367] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
W20240507 08:12:16.859836 5378 leader_election.cc:341] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.859944 5841 ts_tablet_manager.cc:1875] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.860522 5840 raft_consensus.cc:348] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.860687 5840 raft_consensus.cc:374] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.860738 5840 raft_consensus.cc:729] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.860934 5840 consensus_queue.cc:260] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.861227 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2cddcd0d1f884aee9a66a52573eb6011. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.861644 5840 ts_tablet_manager.cc:1398] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:16.862134 6513 ts_tablet_manager.cc:1888] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.862221 6513 log.cc:1196] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/dc74342c8aee4017a212aa88a7f3dc06
I20240507 08:12:16.862555 6513 ts_tablet_manager.cc:1909] T dc74342c8aee4017a212aa88a7f3dc06 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.868019 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534708f1abd4a33965eea92bdc0d0de" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:16.868402 5718 raft_consensus.cc:2454] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:16.868727 5841 ts_tablet_manager.cc:1888] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.868983 5841 log.cc:1196] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b1fadbfd330b4e699e151f83125da281
I20240507 08:12:16.854003 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_089a4f3cc7ad43499020b5c9d1e6f969" } modify_external_catalogs: true
I20240507 08:12:16.798914 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.799751 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 167ee9cbc9174b82824c468e098472a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6563c42d240b4e8685dfa4972f2b9924) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.800485 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet c534708f1abd4a33965eea92bdc0d0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.869350 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_089a4f3cc7ad43499020b5c9d1e6f969" } modify_external_catalogs: true
I20240507 08:12:16.812474 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 982fb83171db41ea9d1572d68e830778 (DEFAULT_TABLE table=default.Table_8f670d8a09204a8fb9319c3d39bd175f [id=900e33cbb529457dbebfb81e176aec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.814066 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 14944e46cb1c4460b1e2f154a8e8a085 (DEFAULT_TABLE table=default.Table_5e6b208d88324e1d913e9c07d26d71e4 [id=975d5db97b8c452692a4ff484b1e9687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.869829 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14944e46cb1c4460b1e2f154a8e8a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.870123 5840 tablet_bootstrap.cc:492] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.854470 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_108b5f297f7f42a0a09409c1a3e4b975" } modify_external_catalogs: true
I20240507 08:12:16.870651 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_108b5f297f7f42a0a09409c1a3e4b975" } modify_external_catalogs: true
I20240507 08:12:16.871436 5840 tablet_bootstrap.cc:654] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.856226 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_558ec8354d9943f6ad3e2be4108aefc6" } modify_external_catalogs: true
I20240507 08:12:16.871585 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_558ec8354d9943f6ad3e2be4108aefc6" } modify_external_catalogs: true
I20240507 08:12:16.872599 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1039) took 4887 ms. Trace:
I20240507 08:12:16.872671 6513 rpcz_store.cc:276] 0507 08:12:11.984944 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:11.984998 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:16.807094 (+4822096us) ts_tablet_manager.cc:1120] Deleting tablet dc74342c8aee4017a212aa88a7f3dc06
0507 08:12:16.807102 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.843311 (+ 36209us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.862055 (+ 18744us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.872582 (+ 10527us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4821886}
I20240507 08:12:16.860998 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 (table default.Table_54a33e3ca40248028587733beb4f6743 [id=9b06744f353d42a69ea7f5b5b60f3a52]) successfully deleted
I20240507 08:12:16.872936 6513 tablet_replica.cc:330] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:16.873075 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 497ab12edc724eef8d6e7b631662a4ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 497ab12edc724eef8d6e7b631662a4ab already in progress: creating tablet
I20240507 08:12:16.873107 6513 raft_consensus.cc:2229] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.873252 6513 raft_consensus.cc:2258] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.815289 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f (DEFAULT_TABLE table=default.Table_54152e52f0df4b3281c833102a7d9aff [id=55e6acafaa384f6795e99cdbcf0c8731]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.873809 6513 ts_tablet_manager.cc:1875] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.798221 5843 tablet_bootstrap.cc:492] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:16.852743 7378 raft_consensus.cc:482] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:16.874058 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f353aa8817c4b1ab5ebd82f6abcfc8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.874011 7378 raft_consensus.cc:504] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.874612 7378 leader_election.cc:288] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.869115 5377 leader_election.cc:302] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.875312 7378 raft_consensus.cc:2781] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:16.875385 7378 raft_consensus.cc:482] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:16.875452 7378 raft_consensus.cc:3037] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.816375 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet a68badb3b5644ed8b007b2b04bc4f6e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.877866 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0542325c80241fca7e317cda64302a9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:16.878281 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 982fb83171db41ea9d1572d68e830778. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.878279 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0542325c80241fca7e317cda64302a9" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:16.878477 5377 leader_election.cc:341] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:16.878819 5378 leader_election.cc:341] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.878913 5378 leader_election.cc:302] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:16.879444 7497 raft_consensus.cc:2726] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:16.880131 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 219bad2b972c45a8930d6f47ca361858
I20240507 08:12:16.880414 7378 raft_consensus.cc:504] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.880949 7378 leader_election.cc:288] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:16.881307 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 219bad2b972c45a8930d6f47ca361858 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.881657 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534708f1abd4a33965eea92bdc0d0de" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:16.881822 5718 raft_consensus.cc:3037] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:16.883598 5843 ts_tablet_manager.cc:1367] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.383s user 0.005s sys 0.010s
I20240507 08:12:16.884569 5718 raft_consensus.cc:2454] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:16.885133 5377 leader_election.cc:302] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:16.885443 7378 raft_consensus.cc:2781] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:16.885879 7378 raft_consensus.cc:686] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:16.885983 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet dc74342c8aee4017a212aa88a7f3dc06 (table default.Table_889601b1d99641d8b8420888cf849ec1 [id=3d6c62194b814cfa97b983900e44f24f]) successfully deleted
I20240507 08:12:16.886176 7378 consensus_queue.cc:237] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.886590 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 219bad2b972c45a8930d6f47ca361858 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.889292 5258 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 6bb74a107e3e472e89f43cf23207d162
I20240507 08:12:16.889808 5843 raft_consensus.cc:348] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.890074 5843 raft_consensus.cc:374] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.890126 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 219bad2b972c45a8930d6f47ca361858 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.890336 5843 raft_consensus.cc:729] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.890812 5843 consensus_queue.cc:260] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.891165 6513 ts_tablet_manager.cc:1888] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:16.891268 6513 log.cc:1196] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/ba7c1aaf55d74c979ec1fc0a57f15f44
I20240507 08:12:16.891618 6513 ts_tablet_manager.cc:1909] T ba7c1aaf55d74c979ec1fc0a57f15f44 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.893218 5843 ts_tablet_manager.cc:1398] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:12:16.894232 5698 tablet_service.cc:1467] Processing CreateTablet for tablet fe8e4d53d3b44387b5ca12d5473947be (DEFAULT_TABLE table=default.Table_9d3174b2f05d48bcae3a5624a768b09a [id=e30f4b8dd1004b6cb23c5ec60f9f249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.894606 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8e4d53d3b44387b5ca12d5473947be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.895635 5843 tablet_bootstrap.cc:492] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:16.893967 5417 tablet_service.cc:1467] Processing CreateTablet for tablet fe8e4d53d3b44387b5ca12d5473947be (DEFAULT_TABLE table=default.Table_9d3174b2f05d48bcae3a5624a768b09a [id=e30f4b8dd1004b6cb23c5ec60f9f249e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.896898 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe8e4d53d3b44387b5ca12d5473947be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.897821 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7fe51f02bf435291ed4dd5ec953597 (DEFAULT_TABLE table=default.Table_1e121ea477624cf8a2fea778d5ca71ad [id=7c709211a46f477b9d7a9e2765789ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.898185 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7fe51f02bf435291ed4dd5ec953597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.899242 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 14944e46cb1c4460b1e2f154a8e8a085 (DEFAULT_TABLE table=default.Table_5e6b208d88324e1d913e9c07d26d71e4 [id=975d5db97b8c452692a4ff484b1e9687]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.899721 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14944e46cb1c4460b1e2f154a8e8a085. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.899715 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3954a4e0d3104161aad427eb2781d55d
I20240507 08:12:16.901178 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 3954a4e0d3104161aad427eb2781d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.894976 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7fe51f02bf435291ed4dd5ec953597 (DEFAULT_TABLE table=default.Table_1e121ea477624cf8a2fea778d5ca71ad [id=7c709211a46f477b9d7a9e2765789ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.901890 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7fe51f02bf435291ed4dd5ec953597. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.903661 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 3954a4e0d3104161aad427eb2781d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.906678 5841 ts_tablet_manager.cc:1909] T b1fadbfd330b4e699e151f83125da281 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.907163 5561 tablet_service.cc:1514] Processing DeleteTablet for tablet 3954a4e0d3104161aad427eb2781d55d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.907938 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 (DEFAULT_TABLE table=default.Table_ee96356f684941cc995051508593ecd8 [id=7bf669127785463eaf97487c8b1993cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.908289 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.908349 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 1c7fe51f02bf435291ed4dd5ec953597 (DEFAULT_TABLE table=default.Table_1e121ea477624cf8a2fea778d5ca71ad [id=7c709211a46f477b9d7a9e2765789ead]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.909188 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1089) took 3327 ms. Trace:
I20240507 08:12:16.909415 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c7fe51f02bf435291ed4dd5ec953597. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.909703 5656 consensus_peers.cc:485] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84212d6cb2c742c1a0f3fdae8be8ebe1. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:16.909469 5841 rpcz_store.cc:276] 0507 08:12:13.582036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.605064 (+ 23028us) service_pool.cc:225] Handling call
0507 08:12:16.859127 (+3254063us) ts_tablet_manager.cc:1120] Deleting tablet b1fadbfd330b4e699e151f83125da281
0507 08:12:16.859134 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.863271 (+ 4137us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.868428 (+ 5157us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.909081 (+ 40653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3253900}
I20240507 08:12:16.910216 5843 tablet_bootstrap.cc:654] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.910480 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b1fadbfd330b4e699e151f83125da281 (table default.Table_307c9baefe434bb68c8d0906a9b9788e [id=6415c5a6c29747b6b005de15fb2ab365]) successfully deleted
I20240507 08:12:16.911342 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1094) took 3303 ms. Trace:
I20240507 08:12:16.911563 5841 rpcz_store.cc:276] 0507 08:12:13.607977 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.608928 (+ 951us) service_pool.cc:225] Handling call
0507 08:12:16.910946 (+3302018us) ts_tablet_manager.cc:1120] Deleting tablet 52e9c1158e27465280e1043d91917e3e
0507 08:12:16.910959 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.911240 (+ 281us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3301752}
W20240507 08:12:16.912320 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 52e9c1158e27465280e1043d91917e3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52e9c1158e27465280e1043d91917e3e already in progress: creating tablet
I20240507 08:12:16.912775 5841 tablet_replica.cc:330] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.913861 5841 raft_consensus.cc:2229] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.914785 5841 raft_consensus.cc:2258] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.916596 5841 ts_tablet_manager.cc:1875] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.919883 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 (DEFAULT_TABLE table=default.Table_ee96356f684941cc995051508593ecd8 [id=7bf669127785463eaf97487c8b1993cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.920236 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.921020 5260 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:16.921401 5261 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3] alter complete (version 3)
I20240507 08:12:16.926460 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0a23d6a405694a7582422d77797c40ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.927210 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a23d6a405694a7582422d77797c40ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.929394 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 (DEFAULT_TABLE table=default.Table_ee96356f684941cc995051508593ecd8 [id=7bf669127785463eaf97487c8b1993cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.929939 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.931479 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet c534708f1abd4a33965eea92bdc0d0de with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:16.934840 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1041) took 4667 ms. Trace:
I20240507 08:12:16.935345 5843 tablet_bootstrap.cc:492] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:16.935496 5843 ts_tablet_manager.cc:1367] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
I20240507 08:12:16.936192 5843 raft_consensus.cc:348] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.936594 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6811dd1c0ba44ac6a6bb5ec1a92bde8f" } new_table_name: "default.Table_206b231a053b42fab47d3c3973502c90" modify_external_catalogs: true
I20240507 08:12:16.937505 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_47429eec1f5942329f871ca8dd068b20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.938074 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47429eec1f5942329f871ca8dd068b20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.939729 5378 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 60911.1us
I20240507 08:12:16.934937 6513 rpcz_store.cc:276] 0507 08:12:12.267100 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.267363 (+ 263us) service_pool.cc:225] Handling call
0507 08:12:16.872897 (+4605534us) ts_tablet_manager.cc:1120] Deleting tablet ba7c1aaf55d74c979ec1fc0a57f15f44
0507 08:12:16.872905 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.888249 (+ 15344us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.891082 (+ 2833us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.934823 (+ 43741us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4605290}
I20240507 08:12:16.940359 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1047) took 4372 ms. Trace:
I20240507 08:12:16.940373 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet ba7c1aaf55d74c979ec1fc0a57f15f44 (table default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5 [id=e7abd1e675454712b5cc1b3d891994b1]) successfully deleted
I20240507 08:12:16.940510 5843 raft_consensus.cc:374] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.940640 5843 raft_consensus.cc:729] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.940819 6513 rpcz_store.cc:276] 0507 08:12:12.568257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.568309 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:16.940241 (+4371932us) ts_tablet_manager.cc:1120] Deleting tablet 1578a63b4e3e4743b1057fa69653cbef
0507 08:12:16.940250 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.940344 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4366556}
I20240507 08:12:16.940879 5843 consensus_queue.cc:260] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:16.941468 5843 ts_tablet_manager.cc:1398] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20240507 08:12:16.942019 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1578a63b4e3e4743b1057fa69653cbef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1578a63b4e3e4743b1057fa69653cbef already in progress: creating tablet
I20240507 08:12:16.942462 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1046) took 4374 ms. Trace:
I20240507 08:12:16.945062 6513 rpcz_store.cc:276] 0507 08:12:12.568126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.568624 (+ 498us) service_pool.cc:225] Handling call
0507 08:12:16.942347 (+4373723us) ts_tablet_manager.cc:1120] Deleting tablet 2b14c80b96bb43088e1d1354a1dbdd31
0507 08:12:16.942356 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.942445 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4366156}
W20240507 08:12:16.948854 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2b14c80b96bb43088e1d1354a1dbdd31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b14c80b96bb43088e1d1354a1dbdd31 already in progress: creating tablet
I20240507 08:12:16.951583 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d89e4185229431f9a68f599da4aff34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.942140 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a154769d0f4501837888feba98f33d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:16.954923 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_093ac864eb004297882e700a6de5da51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:16.955561 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5b9ad4cb5d684e6197ffffffe75bc7ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:16.956161 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b9ad4cb5d684e6197ffffffe75bc7ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:16.956534 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_093ac864eb004297882e700a6de5da51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:16.958300 5843 tablet_bootstrap.cc:492] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:16.959798 5843 tablet_bootstrap.cc:654] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.959841 5840 tablet_bootstrap.cc:492] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.960283 5840 ts_tablet_manager.cc:1367] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.090s user 0.009s sys 0.004s
I20240507 08:12:16.961037 5840 raft_consensus.cc:348] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.961294 5840 raft_consensus.cc:374] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.961405 5840 raft_consensus.cc:729] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.961697 5840 consensus_queue.cc:260] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.962283 5840 ts_tablet_manager.cc:1398] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:16.962604 5840 tablet_bootstrap.cc:492] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:16.963825 5840 tablet_bootstrap.cc:654] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:16.964848 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_910aab0a8d6a4249b5e8a0ae88916c9f" } new_table_name: "default.Table_0f27128e96a24adba366b89374022bff" modify_external_catalogs: true
I20240507 08:12:16.960119 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1051) took 4298 ms. Trace:
I20240507 08:12:16.966132 6513 rpcz_store.cc:276] 0507 08:12:12.662049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.695317 (+ 33268us) service_pool.cc:225] Handling call
0507 08:12:16.949202 (+4253885us) ts_tablet_manager.cc:1120] Deleting tablet 5855bc973d354a69a0579d9a76b3f92b
0507 08:12:16.959965 (+ 10763us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.960099 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4253668}
I20240507 08:12:16.966578 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1055) took 4253 ms. Trace:
I20240507 08:12:16.966714 6513 rpcz_store.cc:276] 0507 08:12:12.712804 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.712862 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:16.966466 (+4253604us) ts_tablet_manager.cc:1120] Deleting tablet a3535f1fbfe34f6e95e9895771ae1350
0507 08:12:16.966475 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.966562 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4253416}
I20240507 08:12:16.967093 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1054) took 4257 ms. Trace:
I20240507 08:12:16.967221 6513 rpcz_store.cc:276] 0507 08:12:12.709370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.789186 (+ 79816us) service_pool.cc:225] Handling call
0507 08:12:16.966998 (+4177812us) ts_tablet_manager.cc:1120] Deleting tablet 37a23d112fcd4ea1b0c2f86133e82e32
0507 08:12:16.967007 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.967079 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4177170}
W20240507 08:12:16.967408 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 5855bc973d354a69a0579d9a76b3f92b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5855bc973d354a69a0579d9a76b3f92b already in progress: creating tablet
I20240507 08:12:16.967590 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1056) took 4234 ms. Trace:
W20240507 08:12:16.967679 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a3535f1fbfe34f6e95e9895771ae1350 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3535f1fbfe34f6e95e9895771ae1350 already in progress: creating tablet
I20240507 08:12:16.967705 6513 rpcz_store.cc:276] 0507 08:12:12.733024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.839862 (+106838us) service_pool.cc:225] Handling call
0507 08:12:16.967495 (+4127633us) ts_tablet_manager.cc:1120] Deleting tablet c1932ffe7d37455c8aa6bb83ffd11498
0507 08:12:16.967504 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.967575 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4127419}
I20240507 08:12:16.967934 6513 tablet_replica.cc:330] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.968112 6513 raft_consensus.cc:2229] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:16.968142 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 37a23d112fcd4ea1b0c2f86133e82e32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37a23d112fcd4ea1b0c2f86133e82e32 already in progress: creating tablet
W20240507 08:12:16.968355 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c1932ffe7d37455c8aa6bb83ffd11498 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1932ffe7d37455c8aa6bb83ffd11498 already in progress: creating tablet
I20240507 08:12:16.968420 6513 raft_consensus.cc:2258] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.968951 6513 ts_tablet_manager.cc:1875] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:16.970592 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 01a9165fdc41482cb1f8431ab6479126 consensus state has no leader
I20240507 08:12:16.963102 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1c100102a43598cd1b2cf4f818524 (DEFAULT_TABLE table=default.Table_2753b56891844d019a4fd40839bc3c59 [id=0b08a1662cf941d69f84ff005367d0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.973958 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1c100102a43598cd1b2cf4f818524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.976186 6513 ts_tablet_manager.cc:1888] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.976336 6513 log.cc:1196] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:16.976730 6513 ts_tablet_manager.cc:1909] T 5471c74c33cd42e8aaf47f71f61adff7 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:16.966768 5841 ts_tablet_manager.cc:1888] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:16.978461 5841 log.cc:1196] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/53bc9bf1c04e4015a152cb750184c9ce
I20240507 08:12:16.978801 5841 ts_tablet_manager.cc:1909] T 53bc9bf1c04e4015a152cb750184c9ce P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:16.980259 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1079) took 4038 ms. Trace:
I20240507 08:12:16.980432 6513 rpcz_store.cc:276] 0507 08:12:12.941813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.945764 (+ 3951us) service_pool.cc:225] Handling call
0507 08:12:16.967895 (+4022131us) ts_tablet_manager.cc:1120] Deleting tablet 5471c74c33cd42e8aaf47f71f61adff7
0507 08:12:16.967903 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.972040 (+ 4137us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.976108 (+ 4068us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.980242 (+ 4134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4021894}
I20240507 08:12:16.980968 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 5471c74c33cd42e8aaf47f71f61adff7 (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:12:16.981316 6513 tablet_replica.cc:330] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:16.981549 6513 raft_consensus.cc:2229] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:16.981918 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b43357777b9341f4bbd950ab972819a3" } new_table_name: "default.Table_36650d696e794757a41588fcf93d8b69" modify_external_catalogs: true
I20240507 08:12:16.982641 6513 raft_consensus.cc:2258] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.983587 6513 ts_tablet_manager.cc:1875] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.985381 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1088) took 3403 ms. Trace:
I20240507 08:12:16.986335 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_42a34eaa8a514f349f1a092abfff155f" } new_table_name: "default.Table_679f7455570d4baf85d594df1b5d2405" modify_external_catalogs: true
I20240507 08:12:16.986546 5841 rpcz_store.cc:276] 0507 08:12:13.581929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.629385 (+ 47456us) service_pool.cc:225] Handling call
0507 08:12:16.912637 (+3283252us) ts_tablet_manager.cc:1120] Deleting tablet 53bc9bf1c04e4015a152cb750184c9ce
0507 08:12:16.912645 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.957345 (+ 44700us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.966618 (+ 9273us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:16.985362 (+ 18744us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3283005}
I20240507 08:12:16.987087 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 53bc9bf1c04e4015a152cb750184c9ce (table default.Table_5b668756389c48ef943681187557167a [id=5331470f800a425eb58c185aef9bd183]) successfully deleted
I20240507 08:12:16.987519 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1095) took 3360 ms. Trace:
I20240507 08:12:16.989509 5840 tablet_bootstrap.cc:492] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:16.926445 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:16.989526 5841 rpcz_store.cc:276] 0507 08:12:13.627131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.648606 (+ 21475us) service_pool.cc:225] Handling call
0507 08:12:16.987384 (+3338778us) ts_tablet_manager.cc:1120] Deleting tablet 74c9c649ec3a4a3db414b78ef9792423
0507 08:12:16.987394 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.987500 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3332037}
W20240507 08:12:16.990069 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 74c9c649ec3a4a3db414b78ef9792423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c9c649ec3a4a3db414b78ef9792423 already in progress: creating tablet
I20240507 08:12:16.990775 5841 tablet_replica.cc:330] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:16.990870 5839 tablet_bootstrap.cc:492] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:16.990993 5841 raft_consensus.cc:2229] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:16.991046 5839 ts_tablet_manager.cc:1367] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.161s user 0.001s sys 0.002s
I20240507 08:12:16.991205 5841 raft_consensus.cc:2258] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:16.991739 5839 raft_consensus.cc:348] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.991992 5839 raft_consensus.cc:374] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:16.992101 5839 raft_consensus.cc:729] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:16.992107 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c534708f1abd4a33965eea92bdc0d0de" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:16.992826 5839 consensus_queue.cc:260] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:16.992844 5548 tablet_service.cc:1467] Processing CreateTablet for tablet f71bab10f45949ed8601ed769242ff90 (DEFAULT_TABLE table=default.Table_9862857aff3b4392b5c75945a0969cf4 [id=fb9fd66c99df452a83375650871520a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.993518 5840 ts_tablet_manager.cc:1367] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.001s
I20240507 08:12:16.993952 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1c100102a43598cd1b2cf4f818524 (DEFAULT_TABLE table=default.Table_2753b56891844d019a4fd40839bc3c59 [id=0b08a1662cf941d69f84ff005367d0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.994254 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1c100102a43598cd1b2cf4f818524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.993386 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bab10f45949ed8601ed769242ff90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:16.992861 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 1ab1c100102a43598cd1b2cf4f818524 (DEFAULT_TABLE table=default.Table_2753b56891844d019a4fd40839bc3c59 [id=0b08a1662cf941d69f84ff005367d0f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.994792 5421 tablet_service.cc:1467] Processing CreateTablet for tablet f71bab10f45949ed8601ed769242ff90 (DEFAULT_TABLE table=default.Table_9862857aff3b4392b5c75945a0969cf4 [id=fb9fd66c99df452a83375650871520a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:16.995024 5378 leader_election.cc:341] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:16.995114 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bab10f45949ed8601ed769242ff90. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:16.995503 5378 consensus_peers.cc:485] T eb1901ddec07495abba085e7188c00d0 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:16.996127 5841 ts_tablet_manager.cc:1875] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:16.997987 5682 tablet_service.cc:1467] Processing CreateTablet for tablet f71bab10f45949ed8601ed769242ff90 (DEFAULT_TABLE table=default.Table_9862857aff3b4392b5c75945a0969cf4 [id=fb9fd66c99df452a83375650871520a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:16.998315 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f71bab10f45949ed8601ed769242ff90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.007309 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa45acab9e48f1ab6418df70f72431 (DEFAULT_TABLE table=default.Table_0d2215c8f3fd4c46ab9709243c307fac [id=259c20ca2ad64a54bcf0d419c4badc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.007688 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa45acab9e48f1ab6418df70f72431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.009043 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5dc60f54a042838832b1800fa03e65 (DEFAULT_TABLE table=default.Table_4ea29887ff37462fa4e6f2ca98542433 [id=c57c977f73a441d6b7ba3159b1bfc7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.009521 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5dc60f54a042838832b1800fa03e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.009793 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5dc60f54a042838832b1800fa03e65 (DEFAULT_TABLE table=default.Table_4ea29887ff37462fa4e6f2ca98542433 [id=c57c977f73a441d6b7ba3159b1bfc7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.010102 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5dc60f54a042838832b1800fa03e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.018683 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa45acab9e48f1ab6418df70f72431 (DEFAULT_TABLE table=default.Table_0d2215c8f3fd4c46ab9709243c307fac [id=259c20ca2ad64a54bcf0d419c4badc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.019133 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa45acab9e48f1ab6418df70f72431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.020776 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0a931f4f08a547d4b928d1ccfab56203" } modify_external_catalogs: true
I20240507 08:12:17.020952 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0a931f4f08a547d4b928d1ccfab56203" } modify_external_catalogs: true
I20240507 08:12:17.021848 5843 tablet_bootstrap.cc:492] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.022032 5843 ts_tablet_manager.cc:1367] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20240507 08:12:17.022797 5843 raft_consensus.cc:348] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.023088 5843 raft_consensus.cc:374] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.023197 5843 raft_consensus.cc:729] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.023483 5843 consensus_queue.cc:260] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.023635 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_38b43849501546438b4a99226360030b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.024096 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38b43849501546438b4a99226360030b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.024864 5843 ts_tablet_manager.cc:1398] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240507 08:12:17.027519 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8e03f8655714d309d5c8ae49c4b314b consensus state has no leader
I20240507 08:12:17.027878 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.028074 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 437fafdae750487ca4655d6a97c230d1 consensus state has no leader
W20240507 08:12:17.028329 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.028447 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9fcb6776bc164f07ba3ad8154d10bf36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.029489 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9fcb6776bc164f07ba3ad8154d10bf36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.030167 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b5153ea0c2884e12a0fbd2cb2fb5678a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.030570 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5153ea0c2884e12a0fbd2cb2fb5678a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.032315 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_cc67c2a61fe048a0acc93d29961dcdce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.032739 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc67c2a61fe048a0acc93d29961dcdce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.033453 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e81e95a06efd420bbaec15335ed3e556" } new_table_name: "default.Table_289233799d4e4d55902c35240f10bdb4" modify_external_catalogs: true
W20240507 08:12:17.034658 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cb968d91481464c830133873b16b8e0 consensus state has no leader
I20240507 08:12:17.034611 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d43fc2157a1546e39dc17b5f26047574" } modify_external_catalogs: true
I20240507 08:12:17.035009 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d43fc2157a1546e39dc17b5f26047574" } modify_external_catalogs: true
I20240507 08:12:17.037344 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ab1c100102a43598cd1b2cf4f818524. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.040637 5839 ts_tablet_manager.cc:1398] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.049s user 0.001s sys 0.001s
I20240507 08:12:17.041647 5840 raft_consensus.cc:348] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.041872 5840 raft_consensus.cc:374] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.041922 5840 raft_consensus.cc:729] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.042102 5840 consensus_queue.cc:260] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.042311 7100 raft_consensus.cc:482] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.042407 7100 raft_consensus.cc:504] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.042640 5840 ts_tablet_manager.cc:1398] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.049s user 0.001s sys 0.000s
I20240507 08:12:17.042987 7100 leader_election.cc:288] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.046766 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.046972 5443 raft_consensus.cc:2454] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.047434 5656 leader_election.cc:302] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.047691 7394 raft_consensus.cc:2781] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.047762 7394 raft_consensus.cc:482] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.047821 7394 raft_consensus.cc:3037] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.049448 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.050282 7394 raft_consensus.cc:504] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.050814 7394 leader_election.cc:288] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.051275 6513 ts_tablet_manager.cc:1888] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.051378 6513 log.cc:1196] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/957775c01be94cc081db344c95a81310
W20240507 08:12:17.051657 5654 leader_election.cc:341] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.051743 6513 ts_tablet_manager.cc:1909] T 957775c01be94cc081db344c95a81310 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.051738 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.051940 5439 raft_consensus.cc:3037] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.052408 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5dc60f54a042838832b1800fa03e65 (DEFAULT_TABLE table=default.Table_4ea29887ff37462fa4e6f2ca98542433 [id=c57c977f73a441d6b7ba3159b1bfc7a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.052774 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5dc60f54a042838832b1800fa03e65. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.054347 5439 raft_consensus.cc:2454] T 7db7b8c560f048f4b02febd371aa2a8c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.054890 5656 leader_election.cc:302] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.055442 6675 raft_consensus.cc:2781] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.055557 6675 raft_consensus.cc:686] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.055786 6675 consensus_queue.cc:237] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.060572 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c1b0a2dddb15451a8e6230ffb52d5302
I20240507 08:12:17.061812 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.062055 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.053433 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7db7b8c560f048f4b02febd371aa2a8c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:17.071144 5654 leader_election.cc:341] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.071738 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet c1b0a2dddb15451a8e6230ffb52d5302 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.076587 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e17502bb4ac64f6ca9c77a3b00ab9f4a
I20240507 08:12:17.078639 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.079387 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.079896 5843 tablet_bootstrap.cc:492] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.080410 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet e17502bb4ac64f6ca9c77a3b00ab9f4a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.081351 5843 tablet_bootstrap.cc:654] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.084082 5840 tablet_bootstrap.cc:492] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.085422 5840 tablet_bootstrap.cc:654] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.087718 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35e387a8fca2455ca6def29673fe2449" } modify_external_catalogs: true
I20240507 08:12:17.087942 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_35e387a8fca2455ca6def29673fe2449" } modify_external_catalogs: true
I20240507 08:12:17.090282 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1080) took 4148 ms. Trace:
I20240507 08:12:17.090476 6513 rpcz_store.cc:276] 0507 08:12:12.941954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.946021 (+ 4067us) service_pool.cc:225] Handling call
0507 08:12:16.981273 (+4035252us) ts_tablet_manager.cc:1120] Deleting tablet 957775c01be94cc081db344c95a81310
0507 08:12:16.981283 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:16.988693 (+ 7410us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.051174 (+ 62481us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.090256 (+ 39082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4035120}
I20240507 08:12:17.091051 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1081) took 4148 ms. Trace:
W20240507 08:12:17.091092 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06e60e00d8474ddb9549a1b6fa209091 consensus state has no leader
I20240507 08:12:17.091178 6513 rpcz_store.cc:276] 0507 08:12:12.942829 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.946163 (+ 3334us) service_pool.cc:225] Handling call
0507 08:12:17.090946 (+4144783us) ts_tablet_manager.cc:1120] Deleting tablet 7841f5ebf5ed4f6eb28dcbd4087f1713
0507 08:12:17.090959 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.091038 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4144653}
I20240507 08:12:17.091679 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 957775c01be94cc081db344c95a81310 (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
I20240507 08:12:17.092425 6513 tablet_replica.cc:330] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:17.092823 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7841f5ebf5ed4f6eb28dcbd4087f1713 because the tablet was not found. No further retry: Not found: Tablet not found: 7841f5ebf5ed4f6eb28dcbd4087f1713
I20240507 08:12:17.093264 6513 raft_consensus.cc:2229] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.093770 6513 raft_consensus.cc:2258] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.094026 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb648fe98e74248b5cd65603de58a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.094571 6513 ts_tablet_manager.cc:1875] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.099599 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0ab8cb559bbe4154a11058e94d2ad349" } modify_external_catalogs: true
I20240507 08:12:17.099846 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0ab8cb559bbe4154a11058e94d2ad349" } modify_external_catalogs: true
I20240507 08:12:17.101317 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 25aa45acab9e48f1ab6418df70f72431 (DEFAULT_TABLE table=default.Table_0d2215c8f3fd4c46ab9709243c307fac [id=259c20ca2ad64a54bcf0d419c4badc31]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.101539 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30e1980aaabd40d4a156b14a39680f90" } modify_external_catalogs: true
I20240507 08:12:17.101707 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_30e1980aaabd40d4a156b14a39680f90" } modify_external_catalogs: true
I20240507 08:12:17.108036 5839 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.112612 5841 ts_tablet_manager.cc:1888] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.116372 5840 tablet_bootstrap.cc:492] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.116590 5840 ts_tablet_manager.cc:1367] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20240507 08:12:17.117023 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25aa45acab9e48f1ab6418df70f72431. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.117465 5840 raft_consensus.cc:348] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.117733 5840 raft_consensus.cc:374] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.117830 5840 raft_consensus.cc:729] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.117903 5843 tablet_bootstrap.cc:492] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.118147 5843 ts_tablet_manager.cc:1367] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.001s
I20240507 08:12:17.118095 5840 consensus_queue.cc:260] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.118880 5840 ts_tablet_manager.cc:1398] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:17.118808 5843 raft_consensus.cc:348] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.119062 5843 raft_consensus.cc:374] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.119165 5843 raft_consensus.cc:729] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.119171 5840 tablet_bootstrap.cc:492] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.119426 5843 consensus_queue.cc:260] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.120414 5840 tablet_bootstrap.cc:654] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.122792 5843 ts_tablet_manager.cc:1398] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240507 08:12:17.123749 5843 tablet_bootstrap.cc:492] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.125412 6513 ts_tablet_manager.cc:1888] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.126581 5843 tablet_bootstrap.cc:654] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.127118 5841 log.cc:1196] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b8ebbcf0d5a54d63a96a9374e839af83
I20240507 08:12:17.129103 5841 ts_tablet_manager.cc:1909] T b8ebbcf0d5a54d63a96a9374e839af83 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.129925 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1099) took 3474 ms. Trace:
I20240507 08:12:17.130219 5841 rpcz_store.cc:276] 0507 08:12:13.655693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.655768 (+ 75us) service_pool.cc:225] Handling call
0507 08:12:16.990731 (+3334963us) ts_tablet_manager.cc:1120] Deleting tablet b8ebbcf0d5a54d63a96a9374e839af83
0507 08:12:16.990740 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.096913 (+106173us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.112495 (+ 15582us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.129906 (+ 17411us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3334815}
I20240507 08:12:17.132186 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b8ebbcf0d5a54d63a96a9374e839af83 (table default.Table_0f7557ddeec745f59ac90d4085a9015c [id=2a51124d3ab54fc58394e23a6aead27c]) successfully deleted
I20240507 08:12:17.137760 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1098) took 3483 ms. Trace:
I20240507 08:12:17.137923 5841 rpcz_store.cc:276] 0507 08:12:13.654244 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.658717 (+ 4473us) service_pool.cc:225] Handling call
0507 08:12:17.132506 (+3473789us) ts_tablet_manager.cc:1120] Deleting tablet 9cf92632da4643cfab913cd925fccef9
0507 08:12:17.137620 (+ 5114us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.137739 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3473612}
W20240507 08:12:17.138478 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9cf92632da4643cfab913cd925fccef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf92632da4643cfab913cd925fccef9 already in progress: creating tablet
I20240507 08:12:17.138929 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1102) took 3456 ms. Trace:
I20240507 08:12:17.139103 5841 rpcz_store.cc:276] 0507 08:12:13.682967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.732904 (+ 49937us) service_pool.cc:225] Handling call
0507 08:12:17.138820 (+3405916us) ts_tablet_manager.cc:1120] Deleting tablet c8aaabc3e26140fda6f86ce0e518b8f5
0507 08:12:17.138829 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.138913 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3397838}
W20240507 08:12:17.139593 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c8aaabc3e26140fda6f86ce0e518b8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8aaabc3e26140fda6f86ce0e518b8f5 already in progress: creating tablet
I20240507 08:12:17.139964 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1105) took 3395 ms. Trace:
I20240507 08:12:17.140074 5841 rpcz_store.cc:276] 0507 08:12:13.744258 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.744471 (+ 213us) service_pool.cc:225] Handling call
0507 08:12:17.139863 (+3395392us) ts_tablet_manager.cc:1120] Deleting tablet d07221497ce7468fb86c257492cef397
0507 08:12:17.139871 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.139948 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3395143}
W20240507 08:12:17.140548 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d07221497ce7468fb86c257492cef397 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d07221497ce7468fb86c257492cef397 already in progress: creating tablet
I20240507 08:12:17.140910 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1116) took 3296 ms. Trace:
I20240507 08:12:17.141021 5841 rpcz_store.cc:276] 0507 08:12:13.844285 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.844331 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:17.140817 (+3296486us) ts_tablet_manager.cc:1120] Deleting tablet 4c62499ccee34036b23e4fe0804fd2ae
0507 08:12:17.140825 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.140895 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3296244}
W20240507 08:12:17.141424 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4c62499ccee34036b23e4fe0804fd2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c62499ccee34036b23e4fe0804fd2ae already in progress: creating tablet
I20240507 08:12:17.141834 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1121) took 2936 ms. Trace:
I20240507 08:12:17.141945 5841 rpcz_store.cc:276] 0507 08:12:14.204992 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.205052 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:17.141728 (+2936676us) ts_tablet_manager.cc:1120] Deleting tablet 2d913dc032b44a4fbab59b42efcba58c
0507 08:12:17.141739 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.141817 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2936504}
W20240507 08:12:17.142372 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 2d913dc032b44a4fbab59b42efcba58c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d913dc032b44a4fbab59b42efcba58c already in progress: creating tablet
I20240507 08:12:17.142717 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1122) took 2936 ms. Trace:
I20240507 08:12:17.142915 5841 rpcz_store.cc:276] 0507 08:12:14.206703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.206749 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:17.142627 (+2935878us) ts_tablet_manager.cc:1120] Deleting tablet 7e26c7ef6f2845cabe09d09045d4feda
0507 08:12:17.142636 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.142703 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2935717}
W20240507 08:12:17.143306 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 7e26c7ef6f2845cabe09d09045d4feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e26c7ef6f2845cabe09d09045d4feda already in progress: creating tablet
I20240507 08:12:17.143640 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1123) took 2879 ms. Trace:
I20240507 08:12:17.143846 5841 rpcz_store.cc:276] 0507 08:12:14.264203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.264269 (+ 66us) service_pool.cc:225] Handling call
0507 08:12:17.143556 (+2879287us) ts_tablet_manager.cc:1120] Deleting tablet 11e779c5c97c44e5bcc7c0eb0194fd95
0507 08:12:17.143564 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.143627 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2879129}
W20240507 08:12:17.144363 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e779c5c97c44e5bcc7c0eb0194fd95 already in progress: creating tablet
I20240507 08:12:17.144696 5841 tablet_replica.cc:330] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.145090 5841 raft_consensus.cc:2229] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.145694 6513 log.cc:1196] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:12:17.145951 5841 raft_consensus.cc:2258] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.142050 5840 tablet_bootstrap.cc:492] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.147220 5841 ts_tablet_manager.cc:1875] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.147392 5840 ts_tablet_manager.cc:1367] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240507 08:12:17.148375 5840 raft_consensus.cc:348] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.148691 5840 raft_consensus.cc:374] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.148833 5840 raft_consensus.cc:729] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.149178 5840 consensus_queue.cc:260] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.149448 5843 tablet_bootstrap.cc:492] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.149766 5843 ts_tablet_manager.cc:1367] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.001s
I20240507 08:12:17.150336 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_118c82c5ecc644d998c3d38c16544c42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.150478 5843 raft_consensus.cc:348] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.150667 5843 raft_consensus.cc:374] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.150722 5843 raft_consensus.cc:729] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:17.150998 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_118c82c5ecc644d998c3d38c16544c42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.150949 5843 consensus_queue.cc:260] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.151558 5843 ts_tablet_manager.cc:1398] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:17.151804 5843 tablet_bootstrap.cc:492] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.153218 5843 tablet_bootstrap.cc:654] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.154084 5840 ts_tablet_manager.cc:1398] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:17.154440 5840 tablet_bootstrap.cc:492] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.155318 5839 tablet_bootstrap.cc:654] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.155424 7552 raft_consensus.cc:482] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.149930 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bf3bbe27b82248878e70d707911633bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.155539 7552 raft_consensus.cc:504] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:17.155843 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf3bbe27b82248878e70d707911633bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.156179 7552 leader_election.cc:288] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.156491 6513 ts_tablet_manager.cc:1909] T b27cc38d0c4c4083900cd6765a5cf6e6 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.156936 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf88112b0b408e8d24828fc93055ef" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.157637 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf88112b0b408e8d24828fc93055ef" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:17.157909 5718 raft_consensus.cc:2454] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:12:17.158042 5378 leader_election.cc:341] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.158474 5377 leader_election.cc:302] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.158923 7552 raft_consensus.cc:2781] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.159004 7552 raft_consensus.cc:482] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.159065 7552 raft_consensus.cc:3037] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.161897 7552 raft_consensus.cc:504] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.162225 7499 raft_consensus.cc:482] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.162345 7499 raft_consensus.cc:504] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.162917 7499 leader_election.cc:288] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.162999 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf88112b0b408e8d24828fc93055ef" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:17.163735 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb648fe98e74248b5cd65603de58a02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:17.163904 5378 leader_election.cc:341] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.163971 5718 raft_consensus.cc:2454] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:17.164197 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1082) took 4221 ms. Trace:
I20240507 08:12:17.164495 5377 leader_election.cc:302] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.164356 6513 rpcz_store.cc:276] 0507 08:12:12.942938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:12.946296 (+ 3358us) service_pool.cc:225] Handling call
0507 08:12:17.092355 (+4146059us) ts_tablet_manager.cc:1120] Deleting tablet b27cc38d0c4c4083900cd6765a5cf6e6
0507 08:12:17.092367 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.121751 (+ 29384us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.125313 (+ 3562us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.164185 (+ 38872us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4145935}
I20240507 08:12:17.164765 7499 raft_consensus.cc:2781] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.164803 6513 tablet_replica.cc:330] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.164822 7499 raft_consensus.cc:482] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.164875 7499 raft_consensus.cc:3037] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.165045 6513 raft_consensus.cc:2229] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.165439 6513 raft_consensus.cc:2258] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.166221 6513 ts_tablet_manager.cc:1875] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.167410 7499 raft_consensus.cc:504] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.167901 7499 leader_election.cc:288] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.168391 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b27cc38d0c4c4083900cd6765a5cf6e6 (table default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]) successfully deleted
I20240507 08:12:17.168716 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb648fe98e74248b5cd65603de58a02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:17.169610 5378 leader_election.cc:341] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.169557 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb648fe98e74248b5cd65603de58a02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:17.169796 5718 raft_consensus.cc:3037] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.170163 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bb648fe98e74248b5cd65603de58a02" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:17.170634 5378 leader_election.cc:341] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.170819 7499 raft_consensus.cc:482] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.170960 7499 raft_consensus.cc:504] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.171535 7499 leader_election.cc:288] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.172086 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:17.172478 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:17.173033 5377 leader_election.cc:341] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.173395 5718 raft_consensus.cc:2454] T 2bb648fe98e74248b5cd65603de58a02 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:12:17.173501 5378 leader_election.cc:341] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.173657 5378 leader_election.cc:302] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.174047 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fb6e602d297c493fa3c7969a29ea399b" } new_table_name: "default.Table_b5e4b55cf0914d33a568f30399e6a052" modify_external_catalogs: true
I20240507 08:12:17.174100 7499 raft_consensus.cc:2726] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.174557 7552 leader_election.cc:288] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.175136 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "accf88112b0b408e8d24828fc93055ef" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:17.176286 5720 raft_consensus.cc:3037] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.179584 5720 raft_consensus.cc:2454] T accf88112b0b408e8d24828fc93055ef P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:17.180099 5377 leader_election.cc:302] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.180688 5377 leader_election.cc:302] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.181123 7499 raft_consensus.cc:2781] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.181322 7550 raft_consensus.cc:2781] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.181416 7550 raft_consensus.cc:686] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:17.181851 7550 consensus_queue.cc:237] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.182201 5840 tablet_bootstrap.cc:654] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:17.182446 5656 consensus_peers.cc:485] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8ebbcf0d5a54d63a96a9374e839af83. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.184788 7499 raft_consensus.cc:686] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:17.185206 7499 consensus_queue.cc:237] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.197222 7394 raft_consensus.cc:482] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.197363 7394 raft_consensus.cc:504] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.198026 7394 leader_election.cc:288] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:17.198184 5378 consensus_peers.cc:485] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:17.198774 5654 consensus_peers.cc:485] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.198926 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.199098 5443 raft_consensus.cc:2454] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.199143 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.199499 5656 leader_election.cc:302] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:17.199627 5654 leader_election.cc:341] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.199800 7394 raft_consensus.cc:2781] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.199864 7394 raft_consensus.cc:482] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.199916 7394 raft_consensus.cc:3037] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.203471 7394 raft_consensus.cc:504] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.204690 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:17.204802 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ab94b1a09dd4969b928b017229ce4bf" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.204947 5443 raft_consensus.cc:3037] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:17.205246 5654 leader_election.cc:341] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.208199 5443 raft_consensus.cc:2454] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.208756 5656 leader_election.cc:302] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.209057 6675 raft_consensus.cc:2781] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.209143 7394 leader_election.cc:288] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.209211 6675 raft_consensus.cc:686] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.209410 6513 ts_tablet_manager.cc:1888] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.209522 6513 log.cc:1196] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/cd4379b6fded4a52843f55f27ae67121
I20240507 08:12:17.209452 6675 consensus_queue.cc:237] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.209872 6513 ts_tablet_manager.cc:1909] T cd4379b6fded4a52843f55f27ae67121 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.212508 5839 tablet_bootstrap.cc:492] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.212754 5839 ts_tablet_manager.cc:1367] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.003s
I20240507 08:12:17.219369 5839 raft_consensus.cc:348] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.219920 5841 ts_tablet_manager.cc:1888] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:17.220024 5839 raft_consensus.cc:374] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.223664 5841 log.cc:1196] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/364903504aa14d08b050a0cf83b82c71
I20240507 08:12:17.223763 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1083) took 4229 ms. Trace:
I20240507 08:12:17.223934 6513 rpcz_store.cc:276] 0507 08:12:12.994205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.039794 (+ 45589us) service_pool.cc:225] Handling call
0507 08:12:17.164742 (+4124948us) ts_tablet_manager.cc:1120] Deleting tablet cd4379b6fded4a52843f55f27ae67121
0507 08:12:17.164756 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.194548 (+ 29792us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.209311 (+ 14763us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.223740 (+ 14429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4122957}
I20240507 08:12:17.224455 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet cd4379b6fded4a52843f55f27ae67121 (table default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]) successfully deleted
I20240507 08:12:17.224853 6513 tablet_replica.cc:330] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.225060 5841 ts_tablet_manager.cc:1909] T 364903504aa14d08b050a0cf83b82c71 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.225085 6513 raft_consensus.cc:2229] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.225275 6513 raft_consensus.cc:2258] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.226047 6513 ts_tablet_manager.cc:1875] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.227797 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1124) took 2897 ms. Trace:
I20240507 08:12:17.229102 5841 rpcz_store.cc:276] 0507 08:12:14.329931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.330188 (+ 257us) service_pool.cc:225] Handling call
0507 08:12:17.144627 (+2814439us) ts_tablet_manager.cc:1120] Deleting tablet 364903504aa14d08b050a0cf83b82c71
0507 08:12:17.144637 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.179315 (+ 34678us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.219815 (+ 40500us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.227779 (+ 7964us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2814262}
I20240507 08:12:17.231478 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_70c8dabdda144b77a238abeacaa20e6e [id=95b44cbcf26f489c8f4d8e07cc615776]) successfully deleted
I20240507 08:12:17.231894 5841 tablet_replica.cc:330] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.232477 5841 raft_consensus.cc:2229] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.233094 5841 raft_consensus.cc:2258] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.234076 5841 ts_tablet_manager.cc:1875] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.234423 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f3940a8999cb4633b1d82922022afe2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.234972 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3940a8999cb4633b1d82922022afe2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.235232 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fca4a8f0d4e244c0872dd877f3e6aec8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.235679 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fca4a8f0d4e244c0872dd877f3e6aec8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.237190 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ef4a536ffde74983b9026698c5be760b" } new_table_name: "default.Table_b1f21cd7024a494eb2a18d89b199abb2" modify_external_catalogs: true
I20240507 08:12:17.240978 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet fda4b1ceaf0b4f3385258f3703e4461b
I20240507 08:12:17.241065 6513 ts_tablet_manager.cc:1888] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:17.241204 6513 log.cc:1196] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/34adee00b529424b9f274c61ce73b0bc
I20240507 08:12:17.241844 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ebd8a847575e430eb26365ca4cf1928a" } new_table_name: "default.Table_6a6d8aad8c6b4f7c906b7e4cd211da73" modify_external_catalogs: true
I20240507 08:12:17.242386 6513 ts_tablet_manager.cc:1909] T 34adee00b529424b9f274c61ce73b0bc P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.242880 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1088) took 3908 ms. Trace:
I20240507 08:12:17.243634 6513 rpcz_store.cc:276] 0507 08:12:13.334467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.334521 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:17.224801 (+3890280us) ts_tablet_manager.cc:1120] Deleting tablet 34adee00b529424b9f274c61ce73b0bc
0507 08:12:17.224812 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.229365 (+ 4553us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.240954 (+ 11589us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.242853 (+ 1899us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3862352}
I20240507 08:12:17.244087 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]) successfully deleted
I20240507 08:12:17.244472 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1104) took 3640 ms. Trace:
I20240507 08:12:17.244547 6513 rpcz_store.cc:276] 0507 08:12:13.603825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.603870 (+ 45us) service_pool.cc:225] Handling call
0507 08:12:17.244338 (+3640468us) ts_tablet_manager.cc:1120] Deleting tablet 497ab12edc724eef8d6e7b631662a4ab
0507 08:12:17.244350 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.244454 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3640330}
W20240507 08:12:17.244975 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 497ab12edc724eef8d6e7b631662a4ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 497ab12edc724eef8d6e7b631662a4ab already in progress: creating tablet
I20240507 08:12:17.245424 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1105) took 3626 ms. Trace:
I20240507 08:12:17.245503 6513 rpcz_store.cc:276] 0507 08:12:13.618489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.618540 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:17.245193 (+3626653us) ts_tablet_manager.cc:1120] Deleting tablet 52e9c1158e27465280e1043d91917e3e
0507 08:12:17.245201 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.245408 (+ 207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3626464}
I20240507 08:12:17.245851 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:17.245887 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 52e9c1158e27465280e1043d91917e3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52e9c1158e27465280e1043d91917e3e already in progress: creating tablet
I20240507 08:12:17.246170 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1106) took 3616 ms. Trace:
I20240507 08:12:17.246239 6513 rpcz_store.cc:276] 0507 08:12:13.629846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.629907 (+ 61us) service_pool.cc:225] Handling call
0507 08:12:17.246084 (+3616177us) ts_tablet_manager.cc:1120] Deleting tablet 74c9c649ec3a4a3db414b78ef9792423
0507 08:12:17.246092 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.246157 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3616002}
W20240507 08:12:17.246625 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 74c9c649ec3a4a3db414b78ef9792423 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74c9c649ec3a4a3db414b78ef9792423 already in progress: creating tablet
I20240507 08:12:17.246938 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1109) took 3586 ms. Trace:
I20240507 08:12:17.247002 6513 rpcz_store.cc:276] 0507 08:12:13.660305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.660357 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:17.246831 (+3586474us) ts_tablet_manager.cc:1120] Deleting tablet 9cf92632da4643cfab913cd925fccef9
0507 08:12:17.246841 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.246922 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3586287}
W20240507 08:12:17.247197 5378 consensus_peers.cc:485] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4306751d16444ec5948c2ba5831063f4. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:17.247345 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9cf92632da4643cfab913cd925fccef9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9cf92632da4643cfab913cd925fccef9 already in progress: creating tablet
I20240507 08:12:17.247650 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1112) took 3557 ms. Trace:
I20240507 08:12:17.247720 6513 rpcz_store.cc:276] 0507 08:12:13.690539 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.690588 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:17.247555 (+3556967us) ts_tablet_manager.cc:1120] Deleting tablet c8aaabc3e26140fda6f86ce0e518b8f5
0507 08:12:17.247565 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.247636 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3556743}
W20240507 08:12:17.248350 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c8aaabc3e26140fda6f86ce0e518b8f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8aaabc3e26140fda6f86ce0e518b8f5 already in progress: creating tablet
I20240507 08:12:17.248898 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.249269 5840 tablet_bootstrap.cc:492] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.249397 5840 ts_tablet_manager.cc:1367] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.000s
I20240507 08:12:17.250483 5840 raft_consensus.cc:348] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.250756 5840 raft_consensus.cc:374] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.250877 5840 raft_consensus.cc:729] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.251166 5840 consensus_queue.cc:260] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.257128 5841 ts_tablet_manager.cc:1888] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.257088 5839 raft_consensus.cc:729] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.257405 5839 consensus_queue.cc:260] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.257550 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2c2e3e12251543a48671815aaeedc37a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.257614 5841 log.cc:1196] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3f01e3311a8b4553a31914922fb08d13
W20240507 08:12:17.258100 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c2e3e12251543a48671815aaeedc37a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.258539 5839 ts_tablet_manager.cc:1398] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.046s user 0.001s sys 0.001s
I20240507 08:12:17.259054 5839 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.243155 7079 ts_tablet_manager.cc:1888] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.259194 7079 log.cc:1196] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/c707458d1c094d87b2204683b28b49e4
I20240507 08:12:17.259542 7079 ts_tablet_manager.cc:1909] T c707458d1c094d87b2204683b28b49e4 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:17.263170 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d792e2f8ef324c8f907ee92a97ca59c4 consensus state has no leader
I20240507 08:12:17.265254 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ad968e67c91142a99564460bd9f033d4" } new_table_name: "default.Table_4bb242ccb16b4a50bfc2823079ec4ab0" modify_external_catalogs: true
I20240507 08:12:17.267621 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 049483b0533642f2b46d8fa0e9b123cc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.268517 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_616cd80c5f274e8ab4b39e0d82adf728" } modify_external_catalogs: true
I20240507 08:12:17.268700 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_616cd80c5f274e8ab4b39e0d82adf728" } modify_external_catalogs: true
I20240507 08:12:17.268972 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1127) took 1739 ms. Trace:
I20240507 08:12:17.269053 7079 rpcz_store.cc:276] 0507 08:12:15.529050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.542789 (+ 13739us) service_pool.cc:225] Handling call
0507 08:12:16.805908 (+1263119us) ts_tablet_manager.cc:1120] Deleting tablet c707458d1c094d87b2204683b28b49e4
0507 08:12:16.805923 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.008769 (+202846us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.243052 (+234283us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.268954 (+ 25902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1262829}
I20240507 08:12:17.269418 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet c707458d1c094d87b2204683b28b49e4 (table default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]) successfully deleted
I20240507 08:12:17.270454 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1128) took 1602 ms. Trace:
I20240507 08:12:17.270498 5840 ts_tablet_manager.cc:1398] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240507 08:12:17.270524 7079 rpcz_store.cc:276] 0507 08:12:15.668391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.668449 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:17.270310 (+1601861us) ts_tablet_manager.cc:1120] Deleting tablet 62895377a8ac4506a2f4e10039dbdd74
0507 08:12:17.270321 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.270437 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1601671}
W20240507 08:12:17.271016 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 62895377a8ac4506a2f4e10039dbdd74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62895377a8ac4506a2f4e10039dbdd74 already in progress: creating tablet
I20240507 08:12:17.271343 7079 tablet_replica.cc:330] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.271649 7079 raft_consensus.cc:2229] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.271886 7079 raft_consensus.cc:2258] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.272442 7079 ts_tablet_manager.cc:1875] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.276163 5840 tablet_bootstrap.cc:492] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.277179 5839 tablet_bootstrap.cc:654] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.277742 5840 tablet_bootstrap.cc:654] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.278409 5843 tablet_bootstrap.cc:492] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.278620 5843 ts_tablet_manager.cc:1367] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.002s
I20240507 08:12:17.279372 5843 raft_consensus.cc:348] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.279647 5843 raft_consensus.cc:374] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240507 08:12:17.279747 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e2759ebfa0840f8b21d749354768c8d consensus state has no leader
I20240507 08:12:17.279752 5843 raft_consensus.cc:729] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.279989 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet c1b0a2dddb15451a8e6230ffb52d5302 from {username='slave'} at 127.0.0.1:44738
W20240507 08:12:17.281008 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet daa3ee2274004cf09e21aa6861049df7 consensus state has no leader
I20240507 08:12:17.282063 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_54786977c4974775aaa70909567f71fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.282140 5843 consensus_queue.cc:260] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:17.282912 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_54786977c4974775aaa70909567f71fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.285558 5843 ts_tablet_manager.cc:1398] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240507 08:12:17.285914 5843 tablet_bootstrap.cc:492] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:17.288305 5378 consensus_peers.cc:485] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.289692 5843 tablet_bootstrap.cc:654] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.289645 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ec50c8f7242944919aadbff5d3b2a54a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.290184 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec50c8f7242944919aadbff5d3b2a54a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.290033 5841 ts_tablet_manager.cc:1909] T 3f01e3311a8b4553a31914922fb08d13 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.290814 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1126) took 2958 ms. Trace:
I20240507 08:12:17.290940 5841 rpcz_store.cc:276] 0507 08:12:14.332621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.332666 (+ 45us) service_pool.cc:225] Handling call
0507 08:12:17.231843 (+2899177us) ts_tablet_manager.cc:1120] Deleting tablet 3f01e3311a8b4553a31914922fb08d13
0507 08:12:17.231854 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.249251 (+ 17397us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.257029 (+ 7778us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.290794 (+ 33765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899010}
I20240507 08:12:17.291422 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3f01e3311a8b4553a31914922fb08d13 (table default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]) successfully deleted
I20240507 08:12:17.292506 5841 tablet_replica.cc:330] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.292758 5841 raft_consensus.cc:2229] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.293808 5841 raft_consensus.cc:2258] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.294394 5841 ts_tablet_manager.cc:1875] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.278021 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1115) took 3528 ms. Trace:
I20240507 08:12:17.300521 6513 rpcz_store.cc:276] 0507 08:12:13.749931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.750191 (+ 260us) service_pool.cc:225] Handling call
0507 08:12:17.276352 (+3526161us) ts_tablet_manager.cc:1120] Deleting tablet d07221497ce7468fb86c257492cef397
0507 08:12:17.277880 (+ 1528us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.278002 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3525953}
I20240507 08:12:17.300954 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1126) took 3458 ms. Trace:
W20240507 08:12:17.301085 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet d07221497ce7468fb86c257492cef397 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d07221497ce7468fb86c257492cef397 already in progress: creating tablet
I20240507 08:12:17.301041 6513 rpcz_store.cc:276] 0507 08:12:13.842710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:13.842760 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:17.300824 (+3458064us) ts_tablet_manager.cc:1120] Deleting tablet 4c62499ccee34036b23e4fe0804fd2ae
0507 08:12:17.300834 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.300936 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3457888}
I20240507 08:12:17.301352 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1132) took 3092 ms. Trace:
I20240507 08:12:17.301430 6513 rpcz_store.cc:276] 0507 08:12:14.209107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.209467 (+ 360us) service_pool.cc:225] Handling call
0507 08:12:17.301257 (+3091790us) ts_tablet_manager.cc:1120] Deleting tablet 2d913dc032b44a4fbab59b42efcba58c
0507 08:12:17.301265 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.301338 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3091596}
W20240507 08:12:17.301756 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 4c62499ccee34036b23e4fe0804fd2ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c62499ccee34036b23e4fe0804fd2ae already in progress: creating tablet
I20240507 08:12:17.301944 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1133) took 3092 ms. Trace:
W20240507 08:12:17.301970 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 2d913dc032b44a4fbab59b42efcba58c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d913dc032b44a4fbab59b42efcba58c already in progress: creating tablet
I20240507 08:12:17.302012 6513 rpcz_store.cc:276] 0507 08:12:14.209227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.209690 (+ 463us) service_pool.cc:225] Handling call
0507 08:12:17.301839 (+3092149us) ts_tablet_manager.cc:1120] Deleting tablet 7e26c7ef6f2845cabe09d09045d4feda
0507 08:12:17.301848 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.301927 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3092005}
W20240507 08:12:17.302335 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 7e26c7ef6f2845cabe09d09045d4feda because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e26c7ef6f2845cabe09d09045d4feda already in progress: creating tablet
I20240507 08:12:17.302937 7079 ts_tablet_manager.cc:1888] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.304332 7079 log.cc:1196] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/975f22194aff40609a3cfa63b9b3aca2
I20240507 08:12:17.304813 7079 ts_tablet_manager.cc:1909] T 975f22194aff40609a3cfa63b9b3aca2 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:17.305408 6675 raft_consensus.cc:482] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.305554 6675 raft_consensus.cc:504] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.305436 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1134) took 3041 ms. Trace:
I20240507 08:12:17.306515 6675 leader_election.cc:288] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.306370 6513 rpcz_store.cc:276] 0507 08:12:14.264045 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.264098 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:17.303483 (+3039385us) ts_tablet_manager.cc:1120] Deleting tablet 11e779c5c97c44e5bcc7c0eb0194fd95
0507 08:12:17.305333 (+ 1850us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.305422 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3039154}
W20240507 08:12:17.308320 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 11e779c5c97c44e5bcc7c0eb0194fd95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 11e779c5c97c44e5bcc7c0eb0194fd95 already in progress: creating tablet
I20240507 08:12:17.308383 6513 tablet_replica.cc:330] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.308579 6513 raft_consensus.cc:2229] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.308766 6513 raft_consensus.cc:2258] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.309192 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_93d4d0c49c4b4bf1b92f816195fe7068" } modify_external_catalogs: true
I20240507 08:12:17.309311 6513 ts_tablet_manager.cc:1875] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.306857 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.309367 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_93d4d0c49c4b4bf1b92f816195fe7068" } modify_external_catalogs: true
W20240507 08:12:17.309887 5654 leader_election.cc:341] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.310214 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.310490 5443 raft_consensus.cc:2454] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.310909 5656 leader_election.cc:302] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.311333 6675 raft_consensus.cc:2781] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.311484 6675 raft_consensus.cc:482] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.311615 6675 raft_consensus.cc:3037] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.312372 5839 tablet_bootstrap.cc:492] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.312589 5839 ts_tablet_manager.cc:1367] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20240507 08:12:17.313452 5839 raft_consensus.cc:348] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.313733 5839 raft_consensus.cc:374] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.313853 5839 raft_consensus.cc:729] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:17.313907 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11290d416c9b476f8f7fdb22e424df01 consensus state has no leader
I20240507 08:12:17.314254 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4205c8d3d874424599f4c1e843ac9543" } new_table_name: "default.Table_02a149a05852404a912a65ef0c6bc205" modify_external_catalogs: true
I20240507 08:12:17.314280 5839 consensus_queue.cc:260] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.314764 5841 ts_tablet_manager.cc:1888] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.315798 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1129) took 1637 ms. Trace:
I20240507 08:12:17.315939 7079 rpcz_store.cc:276] 0507 08:12:15.677943 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.678091 (+ 148us) service_pool.cc:225] Handling call
0507 08:12:17.271302 (+1593211us) ts_tablet_manager.cc:1120] Deleting tablet 975f22194aff40609a3cfa63b9b3aca2
0507 08:12:17.271311 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.275720 (+ 4409us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.302807 (+ 27087us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.315779 (+ 12972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1593038}
I20240507 08:12:17.318235 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:17.318528 6675 raft_consensus.cc:504] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.319576 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:17.320062 5654 leader_election.cc:341] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.320295 5839 ts_tablet_manager.cc:1398] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240507 08:12:17.320685 5839 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.323741 6675 leader_election.cc:288] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.327651 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet aa094773f3bf418492bcd5b5021fd429 with delete_type TABLET_DATA_DELETED (replaced by tablet 451f566fdf2a40a7a61a2d7ae38fffba) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.329154 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:17.329895 5839 tablet_bootstrap.cc:654] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:17.330982 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 consensus state has no leader
I20240507 08:12:17.332119 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet fda4b1ceaf0b4f3385258f3703e4461b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.336840 5840 tablet_bootstrap.cc:492] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.336997 5840 ts_tablet_manager.cc:1367] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.061s user 0.000s sys 0.002s
I20240507 08:12:17.337800 5840 raft_consensus.cc:348] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.338001 5840 raft_consensus.cc:374] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.338071 5840 raft_consensus.cc:729] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.338311 5840 consensus_queue.cc:260] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.341589 5840 ts_tablet_manager.cc:1398] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:17.315402 5843 tablet_bootstrap.cc:492] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.344988 5843 ts_tablet_manager.cc:1367] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.059s user 0.001s sys 0.002s
I20240507 08:12:17.346043 5841 log.cc:1196] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/055a30bfc61b4ff2be9f2418ca59b030
I20240507 08:12:17.323594 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a77151b173d448aa247317ec6160ce0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.346498 5443 raft_consensus.cc:3037] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.346661 7499 consensus_queue.cc:1035] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:17.347609 5841 ts_tablet_manager.cc:1909] T 055a30bfc61b4ff2be9f2418ca59b030 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.352746 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1125) took 3020 ms. Trace:
I20240507 08:12:17.352955 5841 rpcz_store.cc:276] 0507 08:12:14.332512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.332859 (+ 347us) service_pool.cc:225] Handling call
0507 08:12:17.292461 (+2959602us) ts_tablet_manager.cc:1120] Deleting tablet 055a30bfc61b4ff2be9f2418ca59b030
0507 08:12:17.292471 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.311686 (+ 19215us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.314655 (+ 2969us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.352724 (+ 38069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2959492}
I20240507 08:12:17.353495 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 055a30bfc61b4ff2be9f2418ca59b030 (table default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]) successfully deleted
I20240507 08:12:17.354480 5843 raft_consensus.cc:348] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.354678 5843 raft_consensus.cc:374] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.354743 5843 raft_consensus.cc:729] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.354930 5843 consensus_queue.cc:260] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.355453 5843 ts_tablet_manager.cc:1398] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:12:17.355733 5843 tablet_bootstrap.cc:492] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.356518 5443 raft_consensus.cc:2454] T 2a77151b173d448aa247317ec6160ce0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.356948 5656 leader_election.cc:302] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.357404 6675 raft_consensus.cc:2781] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.359758 5841 tablet_replica.cc:330] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.359941 5841 raft_consensus.cc:2229] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.360976 5843 tablet_bootstrap.cc:654] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.365157 7394 consensus_queue.cc:1035] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:17.365880 5841 raft_consensus.cc:2258] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.366660 5841 ts_tablet_manager.cc:1875] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.369174 5840 tablet_bootstrap.cc:492] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.370662 5840 tablet_bootstrap.cc:654] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.372519 6513 ts_tablet_manager.cc:1888] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:17.390002 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3eb49743134b4f84aa6fdd9760c13504
I20240507 08:12:17.390818 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5481ddbcd31449c4b44afb87a97138bd
I20240507 08:12:17.391041 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.392129 5424 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.392241 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 5481ddbcd31449c4b44afb87a97138bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.392324 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5481ddbcd31449c4b44afb87a97138bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.394436 5839 tablet_bootstrap.cc:492] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.394619 5839 ts_tablet_manager.cc:1367] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.074s user 0.002s sys 0.004s
I20240507 08:12:17.395344 5839 raft_consensus.cc:348] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.395594 5839 raft_consensus.cc:374] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.395704 5839 raft_consensus.cc:729] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.395975 5839 consensus_queue.cc:260] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.399062 6513 log.cc:1196] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/364903504aa14d08b050a0cf83b82c71
I20240507 08:12:17.400764 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 71fcb6a8839840838bedb777ff1cbbcc
I20240507 08:12:17.402292 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 34adee00b529424b9f274c61ce73b0bc with delete_type TABLET_DATA_DELETED (replaced by tablet 4a9b0cde718548baad3ecca46e8000c9) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.402318 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.402918 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.392901 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 5481ddbcd31449c4b44afb87a97138bd with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.391297 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 3eb49743134b4f84aa6fdd9760c13504 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.403741 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 71fcb6a8839840838bedb777ff1cbbcc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.404134 5839 ts_tablet_manager.cc:1398] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:17.404323 7394 raft_consensus.cc:482] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.404453 7394 raft_consensus.cc:504] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.405084 7394 leader_election.cc:288] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.405490 5256 catalog_manager.cc:5484] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:17.405699 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.405876 5443 raft_consensus.cc:2454] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.406011 5839 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.406282 5656 leader_election.cc:302] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.406536 7394 raft_consensus.cc:2781] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.406605 7394 raft_consensus.cc:482] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.406666 7394 raft_consensus.cc:3037] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.406783 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.407359 5839 tablet_bootstrap.cc:654] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:17.407351 5654 leader_election.cc:341] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.409293 7394 raft_consensus.cc:504] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.409684 7358 raft_consensus.cc:482] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.409812 7358 raft_consensus.cc:504] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.410348 7358 leader_election.cc:288] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.410915 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0542325c80241fca7e317cda64302a9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:17.411572 5654 leader_election.cc:341] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.411821 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.411998 5445 raft_consensus.cc:3037] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.412031 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e6454c4a59d4c4cae1681bcae0da2d7" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:17.412463 5654 leader_election.cc:341] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.414618 5445 raft_consensus.cc:2454] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.415153 5656 leader_election.cc:302] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.415488 7358 raft_consensus.cc:2781] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.415591 7358 raft_consensus.cc:686] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.415818 7358 consensus_queue.cc:237] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.417882 7394 leader_election.cc:288] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.422971 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b0542325c80241fca7e317cda64302a9" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:17.423568 5656 leader_election.cc:341] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:12:17.423676 5656 leader_election.cc:302] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.424089 7100 raft_consensus.cc:2726] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.433729 5843 tablet_bootstrap.cc:492] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.433833 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 5855bc973d354a69a0579d9a76b3f92b from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:17.433952 5843 ts_tablet_manager.cc:1367] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.004s
I20240507 08:12:17.434703 7394 raft_consensus.cc:482] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.434738 5843 raft_consensus.cc:348] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.434896 7394 raft_consensus.cc:504] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.435125 5843 raft_consensus.cc:374] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.435359 5843 raft_consensus.cc:729] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.423578 6513 ts_tablet_manager.cc:1909] T 364903504aa14d08b050a0cf83b82c71 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.435801 5843 consensus_queue.cc:260] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.436439 7394 leader_election.cc:288] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:17.436587 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1135) took 3106 ms. Trace:
I20240507 08:12:17.436713 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca8c38841db45e38ae5d8d965a16d0e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.436798 5843 ts_tablet_manager.cc:1398] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:17.436961 5443 raft_consensus.cc:2454] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.440519 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8fc4b266d274493fb54ebb06278ac446"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.441227 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fc4b266d274493fb54ebb06278ac446 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.443010 7552 raft_consensus.cc:482] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.443136 7552 raft_consensus.cc:504] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.443598 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_378c906ca8ff4cf199a515389bd939d8" } modify_external_catalogs: true
I20240507 08:12:17.443553 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_19a5c269746d46d7b76ea68c04acd280"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.443720 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_378c906ca8ff4cf199a515389bd939d8" } modify_external_catalogs: true
I20240507 08:12:17.443761 7552 leader_election.cc:288] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:17.444034 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19a5c269746d46d7b76ea68c04acd280 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.444937 5839 tablet_bootstrap.cc:492] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.445231 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.445307 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_010a0902019b4e9381337fc5cd6a3834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.445526 5839 ts_tablet_manager.cc:1367] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.002s
W20240507 08:12:17.445791 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_010a0902019b4e9381337fc5cd6a3834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:17.445912 5378 leader_election.cc:341] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.446940 5839 raft_consensus.cc:348] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.447175 5839 raft_consensus.cc:374] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.447240 5839 raft_consensus.cc:729] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.447466 5839 consensus_queue.cc:260] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.448390 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca8c38841db45e38ae5d8d965a16d0e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.449528 5839 ts_tablet_manager.cc:1398] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:17.449841 5839 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.437558 5656 leader_election.cc:302] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.451583 6513 rpcz_store.cc:276] 0507 08:12:14.330060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.330190 (+ 130us) service_pool.cc:225] Handling call
0507 08:12:17.308328 (+2978138us) ts_tablet_manager.cc:1120] Deleting tablet 364903504aa14d08b050a0cf83b82c71
0507 08:12:17.308337 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.316736 (+ 8399us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.372421 (+ 55685us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.436570 (+ 64149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2977963}
I20240507 08:12:17.452078 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1136) took 3108 ms. Trace:
I20240507 08:12:17.452217 6513 rpcz_store.cc:276] 0507 08:12:14.343731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.343796 (+ 65us) service_pool.cc:225] Handling call
0507 08:12:17.451989 (+3108193us) ts_tablet_manager.cc:1120] Deleting tablet b0b9f85cd78748d3afce83b64b3f823b
0507 08:12:17.451999 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.452063 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3107972}
I20240507 08:12:17.452577 6513 tablet_replica.cc:330] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.452816 6513 raft_consensus.cc:2229] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.453217 6513 raft_consensus.cc:2258] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.453856 6513 ts_tablet_manager.cc:1875] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.454371 5843 tablet_bootstrap.cc:492] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.456149 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a8cdec0cbd5e4a049ec36a0f9c01efe3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.456673 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8cdec0cbd5e4a049ec36a0f9c01efe3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.457739 5843 tablet_bootstrap.cc:654] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.458791 5839 tablet_bootstrap.cc:654] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.460566 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3ef4375925474e789a39a2c28fa6adfe
I20240507 08:12:17.461234 7100 raft_consensus.cc:2781] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.461318 7100 raft_consensus.cc:482] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.461390 7100 raft_consensus.cc:3037] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.461607 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef4375925474e789a39a2c28fa6adfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.461558 5839 tablet_bootstrap.cc:492] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.463321 5839 ts_tablet_manager.cc:1367] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20240507 08:12:17.463896 7100 raft_consensus.cc:504] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.463976 5839 raft_consensus.cc:348] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.464159 5839 raft_consensus.cc:374] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.464543 7100 leader_election.cc:288] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:17.464936 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_289cce254034416eb97d18b1a1e6868f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.465531 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_289cce254034416eb97d18b1a1e6868f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.467093 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 565ac6091f5e48b1b024094d5560e275 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.467592 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef4375925474e789a39a2c28fa6adfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.467864 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6991af2fca624791af1a67498a53b156" } new_table_name: "default.Table_484fa39b3a18453aae9f2cbd61ae46c0" modify_external_catalogs: true
I20240507 08:12:17.442437 5840 tablet_bootstrap.cc:492] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.448930 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:17.449409 5654 consensus_peers.cc:485] T 2d45d93ed0d34806b9d9451c101f10af P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
W20240507 08:12:17.468591 5654 leader_election.cc:341] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.468961 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 364903504aa14d08b050a0cf83b82c71 (table default.Table_70c8dabdda144b77a238abeacaa20e6e [id=95b44cbcf26f489c8f4d8e07cc615776]) successfully deleted
I20240507 08:12:17.469177 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca8c38841db45e38ae5d8d965a16d0e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
W20240507 08:12:17.469343 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b0b9f85cd78748d3afce83b64b3f823b because the tablet was not found. No further retry: Not found: Tablet not found: b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:17.469386 5445 raft_consensus.cc:3037] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.470243 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aca8c38841db45e38ae5d8d965a16d0e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:17.472013 5445 raft_consensus.cc:2454] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.473129 5656 leader_election.cc:302] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.473208 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_eb2b1328f64f4a5d8ebdc6d68d4282a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.473428 7583 raft_consensus.cc:2781] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
W20240507 08:12:17.473723 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb2b1328f64f4a5d8ebdc6d68d4282a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.474035 7583 raft_consensus.cc:686] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.474299 7583 consensus_queue.cc:237] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:17.474601 5377 leader_election.cc:341] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.474701 5377 leader_election.cc:302] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.475348 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ef4375925474e789a39a2c28fa6adfe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:17.475684 5654 leader_election.cc:341] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.476343 7499 raft_consensus.cc:2726] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.476624 5840 ts_tablet_manager.cc:1367] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.003s
I20240507 08:12:17.485981 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2c2e3e12251543a48671815aaeedc37a" } new_table_name: "default.Table_4d6a8373656e4d0da8b3529540c558af" modify_external_catalogs: true
I20240507 08:12:17.486701 5840 raft_consensus.cc:348] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.486898 5840 raft_consensus.cc:374] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.486950 5840 raft_consensus.cc:729] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.487136 5840 consensus_queue.cc:260] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.487253 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_11758edf8d62474b993d1527d5d04ce4" } new_table_name: "default.Table_25848992534e410cb5d2e01cb8287dc2" modify_external_catalogs: true
I20240507 08:12:17.487653 5840 ts_tablet_manager.cc:1398] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.011s user 0.000s sys 0.000s
I20240507 08:12:17.489328 5840 tablet_bootstrap.cc:492] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.464210 5839 raft_consensus.cc:729] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.489997 5839 consensus_queue.cc:260] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.490625 5839 ts_tablet_manager.cc:1398] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20240507 08:12:17.491412 5840 tablet_bootstrap.cc:654] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.491981 5841 ts_tablet_manager.cc:1888] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:17.492108 5841 log.cc:1196] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b0542325c80241fca7e317cda64302a9
I20240507 08:12:17.492413 5841 ts_tablet_manager.cc:1909] T b0542325c80241fca7e317cda64302a9 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.492751 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1127) took 3139 ms. Trace:
I20240507 08:12:17.492838 5841 rpcz_store.cc:276] 0507 08:12:14.353354 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.353404 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:17.359700 (+3006296us) ts_tablet_manager.cc:1120] Deleting tablet b0542325c80241fca7e317cda64302a9
0507 08:12:17.359720 (+ 20us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.388534 (+ 28814us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.491871 (+103337us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.492726 (+ 855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3006124}
I20240507 08:12:17.492837 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bac149077ee54ee3ba57739c3b704a42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.493265 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1128) took 3110 ms. Trace:
I20240507 08:12:17.493347 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b0542325c80241fca7e317cda64302a9 (table default.Table_86003022cb914d9ead81ce912a28d1cd [id=657d76ac3c524ee6988bda523bbd19c3]) successfully deleted
I20240507 08:12:17.493341 5841 rpcz_store.cc:276] 0507 08:12:14.382889 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.382937 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:17.493123 (+3110186us) ts_tablet_manager.cc:1120] Deleting tablet ab350a3f4f3341c2ad0dda3ce054e199
0507 08:12:17.493132 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.493248 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3110016}
W20240507 08:12:17.493346 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac149077ee54ee3ba57739c3b704a42 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:17.493780 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ab350a3f4f3341c2ad0dda3ce054e199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab350a3f4f3341c2ad0dda3ce054e199 already in progress: creating tablet
I20240507 08:12:17.497848 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1129) took 3053 ms. Trace:
I20240507 08:12:17.498173 5841 rpcz_store.cc:276] 0507 08:12:14.444781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.444830 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:17.493545 (+3048715us) ts_tablet_manager.cc:1120] Deleting tablet 55bacb9fd24b46709704eb852e051eb1
0507 08:12:17.493552 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.497829 (+ 4277us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3048548}
I20240507 08:12:17.498790 5839 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.500145 5839 tablet_bootstrap.cc:654] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.501614 6513 ts_tablet_manager.cc:1888] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.501840 6513 log.cc:1196] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:12:17.503100 6513 ts_tablet_manager.cc:1909] T 831153d2b89e43f8b08b086f7a8f3a54 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.503468 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1137) took 3159 ms. Trace:
I20240507 08:12:17.504091 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
I20240507 08:12:17.504249 6513 rpcz_store.cc:276] 0507 08:12:14.343843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.343900 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:17.452532 (+3108632us) ts_tablet_manager.cc:1120] Deleting tablet 831153d2b89e43f8b08b086f7a8f3a54
0507 08:12:17.452541 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.480951 (+ 28410us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.483667 (+ 2716us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.503450 (+ 19783us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3108405}
I20240507 08:12:17.504904 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 831153d2b89e43f8b08b086f7a8f3a54 (table default.Table_1c03c02bf7c44286b5e1273ff2b738c0 [id=4cd92a7dd38946788551d84e24279035]) successfully deleted
I20240507 08:12:17.506183 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1138) took 3162 ms. Trace:
I20240507 08:12:17.507342 6513 rpcz_store.cc:276] 0507 08:12:14.343948 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.344006 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:17.506097 (+3162091us) ts_tablet_manager.cc:1120] Deleting tablet 753019050fcf4dc3b69722c3eb475037
0507 08:12:17.506106 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.506168 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21,"tablet-delete.queue_time_us":3161970}
W20240507 08:12:17.508030 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 753019050fcf4dc3b69722c3eb475037 because the tablet was not found. No further retry: Not found: Tablet not found: 753019050fcf4dc3b69722c3eb475037
I20240507 08:12:17.508455 6513 tablet_replica.cc:330] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
W20240507 08:12:17.508885 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 55bacb9fd24b46709704eb852e051eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55bacb9fd24b46709704eb852e051eb1 already in progress: creating tablet
I20240507 08:12:17.509217 5841 tablet_replica.cc:330] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.508033 5843 tablet_bootstrap.cc:492] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.509061 6513 raft_consensus.cc:2229] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.509475 5841 raft_consensus.cc:2229] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.509491 5843 ts_tablet_manager.cc:1367] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.055s user 0.001s sys 0.003s
I20240507 08:12:17.509865 5841 raft_consensus.cc:2258] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.510203 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
I20240507 08:12:17.510365 5841 ts_tablet_manager.cc:1875] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.510890 5843 raft_consensus.cc:348] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.511128 5843 raft_consensus.cc:374] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.511232 5843 raft_consensus.cc:729] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.510540 5257 catalog_manager.cc:5484] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:17.511654 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (table default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3]). Expected version 2 got 0
I20240507 08:12:17.511442 5843 consensus_queue.cc:260] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.512176 5843 ts_tablet_manager.cc:1398] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240507 08:12:17.512473 5843 tablet_bootstrap.cc:492] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.512969 6513 raft_consensus.cc:2258] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.513916 5839 tablet_bootstrap.cc:492] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.514130 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 19e8ef286cf247278fe2b01218e6310a (DEFAULT_TABLE table=default.Table_bd9f3573c06a41fcbf0fd31c87414464 [id=606e785d00e148c3b76a5cce322ffc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.514165 5839 ts_tablet_manager.cc:1367] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20240507 08:12:17.515815 5839 raft_consensus.cc:348] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.516134 5839 raft_consensus.cc:374] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.516273 5839 raft_consensus.cc:729] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.516597 5839 consensus_queue.cc:260] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.517324 6513 ts_tablet_manager.cc:1875] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.517763 5839 ts_tablet_manager.cc:1398] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:17.518965 5839 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.519073 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]) successfully deleted
I20240507 08:12:17.519660 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e8ef286cf247278fe2b01218e6310a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.521657 7079 tablet_replica.cc:330] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.521865 7079 raft_consensus.cc:2229] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.522020 7079 raft_consensus.cc:2258] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.522588 7079 ts_tablet_manager.cc:1875] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.516762 5840 tablet_bootstrap.cc:492] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.524066 5840 ts_tablet_manager.cc:1367] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.002s
I20240507 08:12:17.524720 5840 raft_consensus.cc:348] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.525135 5840 raft_consensus.cc:374] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.525194 5840 raft_consensus.cc:729] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.525396 5840 consensus_queue.cc:260] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.525950 5840 ts_tablet_manager.cc:1398] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:17.527261 5840 tablet_bootstrap.cc:492] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.528630 5840 tablet_bootstrap.cc:654] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.532434 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7cf76b6dcab2476387df464cd9bb0bcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.532991 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7cf76b6dcab2476387df464cd9bb0bcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:17.533389 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 50358f99145c476fb3b0f8f5e2487aab consensus state has no leader
I20240507 08:12:17.533725 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_26e6e2091f7f4938ba8bdce96b06aeb3" } new_table_name: "default.Table_2c3d31f3a01b41d59815566c10c96fe8" modify_external_catalogs: true
I20240507 08:12:17.535357 5256 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
I20240507 08:12:17.536191 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_028d1e1e16014b8986c0bfebf632a933"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.536793 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_028d1e1e16014b8986c0bfebf632a933 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.537688 7543 raft_consensus.cc:482] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.537817 7543 raft_consensus.cc:504] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.538393 7543 leader_election.cc:288] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.541435 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:17.547283 5656 consensus_peers.cc:485] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b1fadbfd330b4e699e151f83125da281. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:17.547601 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 80c892dab4414411acce4dd160079672 consensus state has no leader
I20240507 08:12:17.550009 7499 consensus_queue.cc:1035] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:17.550941 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03069758ca5945aa9985aa9571ce2bfc has no consensus state
I20240507 08:12:17.551358 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2311302968aa441c8cfc10afe6e08044"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.551951 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2311302968aa441c8cfc10afe6e08044 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.553344 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c0de414a14264debbc99648f2cdabb8f" } new_table_name: "default.Table_e2afc641890242cb9da4ba4a1f94ecc2" modify_external_catalogs: true
I20240507 08:12:17.555637 5841 ts_tablet_manager.cc:1888] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:17.555879 5841 log.cc:1196] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4306751d16444ec5948c2ba5831063f4
I20240507 08:12:17.556135 6513 ts_tablet_manager.cc:1888] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.556264 6513 log.cc:1196] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/33875331045146a9a02559a0b37d5c6f
I20240507 08:12:17.556509 5841 ts_tablet_manager.cc:1909] T 4306751d16444ec5948c2ba5831063f4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.556624 6513 ts_tablet_manager.cc:1909] T 33875331045146a9a02559a0b37d5c6f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:17.557067 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1131) took 2837 ms. Trace:
I20240507 08:12:17.557217 5841 rpcz_store.cc:276] 0507 08:12:14.719946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.719995 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:17.509169 (+2789174us) ts_tablet_manager.cc:1120] Deleting tablet 4306751d16444ec5948c2ba5831063f4
0507 08:12:17.509178 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.531494 (+ 22316us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.555448 (+ 23954us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.557042 (+ 1594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2788984}
I20240507 08:12:17.561003 7079 ts_tablet_manager.cc:1888] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.561563 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1139) took 3217 ms. Trace:
I20240507 08:12:17.562099 6513 rpcz_store.cc:276] 0507 08:12:14.344711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.344749 (+ 38us) service_pool.cc:225] Handling call
0507 08:12:17.508403 (+3163654us) ts_tablet_manager.cc:1120] Deleting tablet 33875331045146a9a02559a0b37d5c6f
0507 08:12:17.508412 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.542241 (+ 33829us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.556041 (+ 13800us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.561540 (+ 5499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3163468}
I20240507 08:12:17.562793 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 33875331045146a9a02559a0b37d5c6f (table default.Table_c2c8b03ca2bf496d8112eb02078a24c4 [id=3273c97f16d6408f9c12e4a04711be33]) successfully deleted
I20240507 08:12:17.563484 6513 tablet_replica.cc:330] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.564637 6513 raft_consensus.cc:2229] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.565297 6513 raft_consensus.cc:2258] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.570166 6513 ts_tablet_manager.cc:1875] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.577538 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]) successfully deleted
I20240507 08:12:17.558917 5843 tablet_bootstrap.cc:654] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.580485 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1133) took 2627 ms. Trace:
I20240507 08:12:17.581717 5841 rpcz_store.cc:276] 0507 08:12:14.952853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.953500 (+ 647us) service_pool.cc:225] Handling call
0507 08:12:17.580336 (+2626836us) ts_tablet_manager.cc:1120] Deleting tablet 093fb3986a674b93a435408c7ec85500
0507 08:12:17.580347 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.580461 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2626590}
I20240507 08:12:17.582046 7079 log.cc:1196] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b6e68364238c4b66b8c0de0f4a2ea855
I20240507 08:12:17.582463 7079 ts_tablet_manager.cc:1909] T b6e68364238c4b66b8c0de0f4a2ea855 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:17.582825 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 093fb3986a674b93a435408c7ec85500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 093fb3986a674b93a435408c7ec85500 already in progress: creating tablet
I20240507 08:12:17.583387 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1134) took 2625 ms. Trace:
I20240507 08:12:17.584770 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1131) took 1896 ms. Trace:
I20240507 08:12:17.584792 6513 ts_tablet_manager.cc:1888] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.584906 6513 log.cc:1196] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:17.584919 5841 rpcz_store.cc:276] 0507 08:12:14.957467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.957564 (+ 97us) service_pool.cc:225] Handling call
0507 08:12:17.583275 (+2625711us) ts_tablet_manager.cc:1120] Deleting tablet 19b77b6ef778454e91105e1e9b60020d
0507 08:12:17.583285 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.583371 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2625301}
W20240507 08:12:17.585323 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 19b77b6ef778454e91105e1e9b60020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b77b6ef778454e91105e1e9b60020d already in progress: creating tablet
I20240507 08:12:17.586777 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1136) took 2549 ms. Trace:
I20240507 08:12:17.586864 5841 rpcz_store.cc:276] 0507 08:12:15.037315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.037402 (+ 87us) service_pool.cc:225] Handling call
0507 08:12:17.586667 (+2549265us) ts_tablet_manager.cc:1120] Deleting tablet 62d6eac6baa249eb847c5625d64f998f
0507 08:12:17.586676 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.586762 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2530280}
W20240507 08:12:17.587234 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 62d6eac6baa249eb847c5625d64f998f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d6eac6baa249eb847c5625d64f998f already in progress: creating tablet
I20240507 08:12:17.587405 7079 rpcz_store.cc:276] 0507 08:12:15.688801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.688856 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:17.521611 (+1832755us) ts_tablet_manager.cc:1120] Deleting tablet b6e68364238c4b66b8c0de0f4a2ea855
0507 08:12:17.521620 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.557915 (+ 36295us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.560920 (+ 3005us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.584750 (+ 23830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832537}
I20240507 08:12:17.587965 5839 tablet_bootstrap.cc:654] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.568454 7543 raft_consensus.cc:482] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.592041 7543 raft_consensus.cc:504] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.592621 7543 leader_election.cc:288] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.594403 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b6e68364238c4b66b8c0de0f4a2ea855 (table default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]) successfully deleted
I20240507 08:12:17.562775 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d45d93ed0d34806b9d9451c101f10af" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.565701 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 28e5a380f2784e3a806afcbb5cd2519a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:03 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.596105 5520 leader_election.cc:302] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.590688 7079 tablet_replica.cc:330] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.596406 7079 raft_consensus.cc:2229] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:17.596494 5520 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 049483b0533642f2b46d8fa0e9b123cc
I20240507 08:12:17.596544 7079 raft_consensus.cc:2258] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.597048 7079 ts_tablet_manager.cc:1875] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.597141 7543 raft_consensus.cc:2726] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.599303 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1141) took 2531 ms. Trace:
I20240507 08:12:17.599443 5841 rpcz_store.cc:276] 0507 08:12:15.067145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.067196 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:17.590554 (+2523358us) ts_tablet_manager.cc:1120] Deleting tablet db92da02e5af49f5a13a814f3ce4f9c3
0507 08:12:17.595366 (+ 4812us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.598943 (+ 3577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2523191}
I20240507 08:12:17.599813 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1140) took 2532 ms. Trace:
I20240507 08:12:17.599889 5841 rpcz_store.cc:276] 0507 08:12:15.067029 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.104508 (+ 37479us) service_pool.cc:225] Handling call
0507 08:12:17.599715 (+2495207us) ts_tablet_manager.cc:1120] Deleting tablet a24813c72abd41c3830c7392c6149151
0507 08:12:17.599724 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.599798 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2494979}
W20240507 08:12:17.599970 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet db92da02e5af49f5a13a814f3ce4f9c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db92da02e5af49f5a13a814f3ce4f9c3 already in progress: creating tablet
I20240507 08:12:17.600174 7550 raft_consensus.cc:482] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.600327 7550 raft_consensus.cc:504] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:17.600466 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a24813c72abd41c3830c7392c6149151 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24813c72abd41c3830c7392c6149151 already in progress: creating tablet
I20240507 08:12:17.600924 7550 leader_election.cc:288] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.602319 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c9a65a8e939f4b7d89d00720845ea1fa
W20240507 08:12:17.602670 5517 leader_election.cc:341] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 049483b0533642f2b46d8fa0e9b123cc
I20240507 08:12:17.602772 5517 leader_election.cc:302] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.603258 7543 raft_consensus.cc:2726] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.603574 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.603909 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee23488ce5a940559de6005aa58093f5" } modify_external_catalogs: true
I20240507 08:12:17.604070 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee23488ce5a940559de6005aa58093f5" } modify_external_catalogs: true
I20240507 08:12:17.604627 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.606426 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f27128e96a24adba366b89374022bff" } modify_external_catalogs: true
I20240507 08:12:17.606570 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f27128e96a24adba366b89374022bff" } modify_external_catalogs: true
I20240507 08:12:17.600186 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1153) took 2394 ms. Trace:
I20240507 08:12:17.608057 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_238daf40648c421cbd0954bcbce05338" } modify_external_catalogs: true
I20240507 08:12:17.608196 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_238daf40648c421cbd0954bcbce05338" } modify_external_catalogs: true
I20240507 08:12:17.608158 5841 rpcz_store.cc:276] 0507 08:12:15.205899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.206118 (+ 219us) service_pool.cc:225] Handling call
0507 08:12:17.600094 (+2393976us) ts_tablet_manager.cc:1120] Deleting tablet cc798eb4e02c456880ef3014ebbb7e24
0507 08:12:17.600102 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.600174 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393812}
I20240507 08:12:17.608520 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1154) took 2378 ms. Trace:
W20240507 08:12:17.608582 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cc798eb4e02c456880ef3014ebbb7e24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc798eb4e02c456880ef3014ebbb7e24 already in progress: creating tablet
I20240507 08:12:17.608588 5841 rpcz_store.cc:276] 0507 08:12:15.230374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.234212 (+ 3838us) service_pool.cc:225] Handling call
0507 08:12:17.608409 (+2374197us) ts_tablet_manager.cc:1120] Deleting tablet f748727d3a88442d90cf81bae48acd36
0507 08:12:17.608417 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.608498 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373934}
I20240507 08:12:17.608825 5841 tablet_replica.cc:330] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.609011 5841 raft_consensus.cc:2229] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:17.609326 5841 raft_consensus.cc:2258] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.609874 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:17.611310 5841 ts_tablet_manager.cc:1875] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.614353 5258 catalog_manager.cc:5484] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:17.614791 5258 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
I20240507 08:12:17.623363 7079 ts_tablet_manager.cc:1888] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:17.623487 7079 log.cc:1196] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/150efae7834740b798af86ae1dcfe445
I20240507 08:12:17.623812 7079 ts_tablet_manager.cc:1909] T 150efae7834740b798af86ae1dcfe445 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:17.624372 5843 tablet_bootstrap.cc:492] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.624543 5843 ts_tablet_manager.cc:1367] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.112s user 0.002s sys 0.001s
I20240507 08:12:17.625428 5841 ts_tablet_manager.cc:1888] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.625363 5843 raft_consensus.cc:348] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.626864 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 19e8ef286cf247278fe2b01218e6310a (DEFAULT_TABLE table=default.Table_bd9f3573c06a41fcbf0fd31c87414464 [id=606e785d00e148c3b76a5cce322ffc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.626897 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 (DEFAULT_TABLE table=default.Table_d2db26a18c77448aba96e29400de518e [id=87ce406f968046f4aacde7d657c86042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.627064 5843 raft_consensus.cc:374] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.627137 5843 raft_consensus.cc:729] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.627236 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e8ef286cf247278fe2b01218e6310a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.627476 7543 raft_consensus.cc:482] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.627588 7543 raft_consensus.cc:504] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.627890 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 05751530548b4738b462e9c3cd9f0dea (DEFAULT_TABLE table=default.Table_f618a8c95a604885884b613b4fdd19c8 [id=bae1e8c5a99d48d098c39cb9896af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.628157 7543 leader_election.cc:288] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.628959 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 00091474fd5c487d96db0d3d61665b81 (DEFAULT_TABLE table=default.Table_02610397b3da426491da22c6ee52dc68 [id=e5d081778ab54cc39d90d00a4d42559a]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.629189 5520 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:12:17.629510 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 400e3f14d7764cc7b36d79037838fb9f (DEFAULT_TABLE table=default.Table_7949df2d12e8461292e82198f29d9367 [id=217cb5c16b614a879da4de2da72bea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.629609 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 3859684aac9b4792975cbc22c95a6775 (DEFAULT_TABLE table=default.Table_442b2e36429540cf9ea03222ff0a0ced [id=4bb4470a406a446c9427eb1f6315e76d]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.630111 5517 leader_election.cc:341] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:12:17.630214 5517 leader_election.cc:302] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.630571 7543 raft_consensus.cc:2726] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.630849 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 00091474fd5c487d96db0d3d61665b81 (DEFAULT_TABLE table=default.Table_02610397b3da426491da22c6ee52dc68 [id=e5d081778ab54cc39d90d00a4d42559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.631204 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00091474fd5c487d96db0d3d61665b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.631943 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d89e4185229431f9a68f599da4aff34" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.632660 5562 tablet_service.cc:1467] Processing CreateTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 (DEFAULT_TABLE table=default.Table_5e6f4cb669d74f3594bba624c848009d [id=78867b72819443e29518ba0b25c73787]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.632685 5378 leader_election.cc:341] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.633278 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00091474fd5c487d96db0d3d61665b81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.633430 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 (DEFAULT_TABLE table=default.Table_d2db26a18c77448aba96e29400de518e [id=87ce406f968046f4aacde7d657c86042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.633566 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 05751530548b4738b462e9c3cd9f0dea (DEFAULT_TABLE table=default.Table_f618a8c95a604885884b613b4fdd19c8 [id=bae1e8c5a99d48d098c39cb9896af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.633769 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9354ceaf6f6d4f529d9dbe7b78360b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.634258 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05751530548b4738b462e9c3cd9f0dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.637765 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400e3f14d7764cc7b36d79037838fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.627368 5843 consensus_queue.cc:260] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.639972 5843 ts_tablet_manager.cc:1398] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240507 08:12:17.641395 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05751530548b4738b462e9c3cd9f0dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.641899 5843 tablet_bootstrap.cc:492] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.645818 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3859684aac9b4792975cbc22c95a6775. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.645818 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 0add560e10e04e7c8507e0d8c4632dab (DEFAULT_TABLE table=default.Table_aac731cdd6804a3da20d6eae6ea0e45b [id=fd73ef5942da4a6c9abb8ed86e807f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.646142 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0add560e10e04e7c8507e0d8c4632dab. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:17.650826 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.059s user 0.002s sys 0.000s
I20240507 08:12:17.651468 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 3859684aac9b4792975cbc22c95a6775 (DEFAULT_TABLE table=default.Table_442b2e36429540cf9ea03222ff0a0ced [id=4bb4470a406a446c9427eb1f6315e76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.652931 6513 ts_tablet_manager.cc:1909] T 85dc58133d2b4f5496b4e570ae6494c0 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:17.653298 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3b065c19b8d486d99689f9d7c7b9c25 consensus state has no leader
I20240507 08:12:17.653766 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9354ceaf6f6d4f529d9dbe7b78360b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.653795 5410 tablet_service.cc:1467] Processing CreateTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 (DEFAULT_TABLE table=default.Table_5e6f4cb669d74f3594bba624c848009d [id=78867b72819443e29518ba0b25c73787]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.654145 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4b78c08d758043c78d5bb8281b7f2d62
I20240507 08:12:17.654451 5840 tablet_bootstrap.cc:492] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.654577 5840 ts_tablet_manager.cc:1367] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.005s
I20240507 08:12:17.654837 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d8e03f8655714d309d5c8ae49c4b314b
I20240507 08:12:17.655218 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.655297 5841 log.cc:1196] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8a015bf84a454d49b6497c16ed15c84a
I20240507 08:12:17.655450 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet fe721ba2f0ed4a3a887e7496e85d163c
I20240507 08:12:17.655226 5840 raft_consensus.cc:348] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.655766 5840 raft_consensus.cc:374] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.655825 5840 raft_consensus.cc:729] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.656057 5840 consensus_queue.cc:260] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.656738 5840 ts_tablet_manager.cc:1398] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:17.656778 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e03f8655714d309d5c8ae49c4b314b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.656877 5692 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e03f8655714d309d5c8ae49c4b314b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.656890 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.657022 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.657090 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.658394 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet d8e03f8655714d309d5c8ae49c4b314b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.658596 5558 tablet_service.cc:1514] Processing DeleteTablet for tablet fe721ba2f0ed4a3a887e7496e85d163c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.660966 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 0add560e10e04e7c8507e0d8c4632dab (DEFAULT_TABLE table=default.Table_aac731cdd6804a3da20d6eae6ea0e45b [id=fd73ef5942da4a6c9abb8ed86e807f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.661439 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 0add560e10e04e7c8507e0d8c4632dab (DEFAULT_TABLE table=default.Table_aac731cdd6804a3da20d6eae6ea0e45b [id=fd73ef5942da4a6c9abb8ed86e807f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.590078 7394 raft_consensus.cc:482] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.601430 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d89e4185229431f9a68f599da4aff34" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:17.662518 5720 raft_consensus.cc:2454] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:17.604053 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet c9a65a8e939f4b7d89d00720845ea1fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.613119 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bb648fe98e74248b5cd65603de58a02 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:05 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.613265 5691 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.613546 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet aca8c38841db45e38ae5d8d965a16d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.626890 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 19e8ef286cf247278fe2b01218e6310a (DEFAULT_TABLE table=default.Table_bd9f3573c06a41fcbf0fd31c87414464 [id=606e785d00e148c3b76a5cce322ffc61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.632092 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 (DEFAULT_TABLE table=default.Table_d2db26a18c77448aba96e29400de518e [id=87ce406f968046f4aacde7d657c86042]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.632393 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 05751530548b4738b462e9c3cd9f0dea (DEFAULT_TABLE table=default.Table_f618a8c95a604885884b613b4fdd19c8 [id=bae1e8c5a99d48d098c39cb9896af050]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.662892 5377 leader_election.cc:302] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.632660 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 00091474fd5c487d96db0d3d61665b81 (DEFAULT_TABLE table=default.Table_02610397b3da426491da22c6ee52dc68 [id=e5d081778ab54cc39d90d00a4d42559a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.632936 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 400e3f14d7764cc7b36d79037838fb9f (DEFAULT_TABLE table=default.Table_7949df2d12e8461292e82198f29d9367 [id=217cb5c16b614a879da4de2da72bea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.632099 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 3859684aac9b4792975cbc22c95a6775 (DEFAULT_TABLE table=default.Table_442b2e36429540cf9ea03222ff0a0ced [id=4bb4470a406a446c9427eb1f6315e76d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.649873 5690 tablet_service.cc:1467] Processing CreateTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 (DEFAULT_TABLE table=default.Table_5e6f4cb669d74f3594bba624c848009d [id=78867b72819443e29518ba0b25c73787]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.663285 7550 raft_consensus.cc:2781] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.663219 7394 raft_consensus.cc:504] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.663368 7550 raft_consensus.cc:482] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.663437 7550 raft_consensus.cc:3037] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.663556 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 400e3f14d7764cc7b36d79037838fb9f (DEFAULT_TABLE table=default.Table_7949df2d12e8461292e82198f29d9367 [id=217cb5c16b614a879da4de2da72bea1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.664158 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:17.664235 7394 leader_election.cc:288] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.664535 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:17.664649 5654 leader_election.cc:341] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.664716 5439 raft_consensus.cc:2454] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.665014 5656 leader_election.cc:302] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.665011 5420 tablet_service.cc:1467] Processing CreateTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 (DEFAULT_TABLE table=default.Table_0dbb7701a8a94990a7d51582422d59ac [id=5abe3632a8a9431088b3b4e1ce8e603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.665680 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3859684aac9b4792975cbc22c95a6775. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.666209 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0dbb7701a8a94990a7d51582422d59ac" } modify_external_catalogs: true
I20240507 08:12:17.666388 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0dbb7701a8a94990a7d51582422d59ac" } modify_external_catalogs: true
I20240507 08:12:17.666378 7550 raft_consensus.cc:504] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.666541 5554 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b78c08d758043c78d5bb8281b7f2d62 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.666858 5561 tablet_service.cc:1467] Processing CreateTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 (DEFAULT_TABLE table=default.Table_0dbb7701a8a94990a7d51582422d59ac [id=5abe3632a8a9431088b3b4e1ce8e603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.667165 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3d1e9e7e53483fa75f7cd918295e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.667623 7550 leader_election.cc:288] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.668663 7358 raft_consensus.cc:2781] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.668745 7358 raft_consensus.cc:482] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.668804 7358 raft_consensus.cc:3037] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.668812 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9bfb20b9d7454bc2b58d6c328b6ecc47"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.669286 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bfb20b9d7454bc2b58d6c328b6ecc47 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.669718 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d89e4185229431f9a68f599da4aff34" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:17.670187 5378 leader_election.cc:341] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.671625 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 5170fb1c78db4149a6ee865b227a5f6f (DEFAULT_TABLE table=default.Table_8dc2055582bf4aa6aa58fa48b51f7b1b [id=7db4ddc87e6b45c6a981afe5535a42d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.671943 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5170fb1c78db4149a6ee865b227a5f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.672360 7358 raft_consensus.cc:504] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.672461 5682 tablet_service.cc:1467] Processing CreateTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 (DEFAULT_TABLE table=default.Table_0dbb7701a8a94990a7d51582422d59ac [id=5abe3632a8a9431088b3b4e1ce8e603e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.673317 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 5170fb1c78db4149a6ee865b227a5f6f (DEFAULT_TABLE table=default.Table_8dc2055582bf4aa6aa58fa48b51f7b1b [id=7db4ddc87e6b45c6a981afe5535a42d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.673425 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 5170fb1c78db4149a6ee865b227a5f6f (DEFAULT_TABLE table=default.Table_8dc2055582bf4aa6aa58fa48b51f7b1b [id=7db4ddc87e6b45c6a981afe5535a42d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.673698 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:17.674132 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7b4eb72b855459b8da46c037108097e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.673854 7358 leader_election.cc:288] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:17.674147 5654 leader_election.cc:341] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.674353 5439 raft_consensus.cc:3037] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.674605 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.675102 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0add560e10e04e7c8507e0d8c4632dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.675956 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 63c2d87c949a4bd18e0541bad7606e02 (DEFAULT_TABLE table=default.Table_093ac864eb004297882e700a6de5da51 [id=e3ae7f4aa7e2403a8f88e14ea0604b59]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.676319 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c2d87c949a4bd18e0541bad7606e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.680280 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400e3f14d7764cc7b36d79037838fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.683754 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3d1e9e7e53483fa75f7cd918295e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.684665 5439 raft_consensus.cc:2454] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.685098 5656 leader_election.cc:302] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.685406 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5170fb1c78db4149a6ee865b227a5f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.685683 7579 raft_consensus.cc:2781] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.685815 7579 raft_consensus.cc:686] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.686053 7579 consensus_queue.cc:237] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.686638 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 63c2d87c949a4bd18e0541bad7606e02 (DEFAULT_TABLE table=default.Table_093ac864eb004297882e700a6de5da51 [id=e3ae7f4aa7e2403a8f88e14ea0604b59]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.688642 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 50358f99145c476fb3b0f8f5e2487aab (table default.Table_484fa39b3a18453aae9f2cbd61ae46c0 [id=910107a4e3854212a3cad0ddb4888b96], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 50358f99145c476fb3b0f8f5e2487aab consensus state has no leader
I20240507 08:12:17.689601 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cc3d1e9e7e53483fa75f7cd918295e50
I20240507 08:12:17.692046 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.692083 5555 tablet_service.cc:1467] Processing CreateTablet for tablet b1c1a5919b5247c8aa542b806611dfe7 (DEFAULT_TABLE table=default.Table_05652d1f4f574b078ce9726ea093c0eb [id=0251e92ad48045dda038e86c52f525da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.692297 7543 raft_consensus.cc:482] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.692425 7543 raft_consensus.cc:504] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.692551 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c1a5919b5247c8aa542b806611dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.693042 7543 leader_election.cc:288] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.693058 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.693246 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7f072f55a94cca938aa063637af9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.693346 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3be7b44034d25ab402f11166e2ab5 (DEFAULT_TABLE table=default.Table_7ef0ec9add454eb28c8bb9725ee5274f [id=e2ed23c303c34e25942822f70772c7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.693466 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet aca8c38841db45e38ae5d8d965a16d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.693715 5416 tablet_service.cc:1467] Processing CreateTablet for tablet b1c1a5919b5247c8aa542b806611dfe7 (DEFAULT_TABLE table=default.Table_05652d1f4f574b078ce9726ea093c0eb [id=0251e92ad48045dda038e86c52f525da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.694964 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1130) took 2006 ms. Trace:
I20240507 08:12:17.695015 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.695058 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7c4a61a066d64809ada26672081b8ed7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.695222 7079 rpcz_store.cc:276] 0507 08:12:15.688656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.689214 (+ 558us) service_pool.cc:225] Handling call
0507 08:12:17.590637 (+1901423us) ts_tablet_manager.cc:1120] Deleting tablet 150efae7834740b798af86ae1dcfe445
0507 08:12:17.590645 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.617175 (+ 26530us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.623275 (+ 6100us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.694946 (+ 71671us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1408,"tablet-delete.queue_time_us":1901263}
I20240507 08:12:17.695597 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 150efae7834740b798af86ae1dcfe445 (table default.Table_1d73a8ee5a4848628ed2472bdc0b084b [id=d95a0f67e809437da87ad837fbf8f2a3]) successfully deleted
I20240507 08:12:17.696174 5692 tablet_service.cc:1467] Processing CreateTablet for tablet b1c1a5919b5247c8aa542b806611dfe7 (DEFAULT_TABLE table=default.Table_05652d1f4f574b078ce9726ea093c0eb [id=0251e92ad48045dda038e86c52f525da]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.696287 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet cc3d1e9e7e53483fa75f7cd918295e50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:17.695592 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c4a61a066d64809ada26672081b8ed7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.696928 7079 tablet_replica.cc:330] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.697095 7079 raft_consensus.cc:2229] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.697213 7079 raft_consensus.cc:2258] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:17.697428 5654 consensus_peers.cc:485] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:17.697696 7079 ts_tablet_manager.cc:1875] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.697815 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 63c2d87c949a4bd18e0541bad7606e02 (DEFAULT_TABLE table=default.Table_093ac864eb004297882e700a6de5da51 [id=e3ae7f4aa7e2403a8f88e14ea0604b59]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.698225 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1935dd9e2a34c49bb3e0021f46bc765 consensus state has no leader
I20240507 08:12:17.698158 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3be7b44034d25ab402f11166e2ab5 (DEFAULT_TABLE table=default.Table_7ef0ec9add454eb28c8bb9725ee5274f [id=e2ed23c303c34e25942822f70772c7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.698578 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d89e4185229431f9a68f599da4aff34 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.698781 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 167ee9cbc9174b82824c468e098472a6 with delete_type TABLET_DATA_DELETED (replaced by tablet 6563c42d240b4e8685dfa4972f2b9924) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.706027 7543 raft_consensus.cc:482] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.706190 7543 raft_consensus.cc:504] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:17.706197 5520 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 31251576beee4a8a81a453b721c9c058
W20240507 08:12:17.706933 5654 consensus_peers.cc:485] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 975f22194aff40609a3cfa63b9b3aca2. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.707566 7543 leader_election.cc:288] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:17.708256 5520 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 565ac6091f5e48b1b024094d5560e275
I20240507 08:12:17.709713 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.652431 6675 raft_consensus.cc:686] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.714706 6675 consensus_queue.cc:237] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.580086 7590 raft_consensus.cc:482] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.718052 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d89e4185229431f9a68f599da4aff34" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:17.718241 5720 raft_consensus.cc:3037] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.720126 5841 ts_tablet_manager.cc:1909] T 8a015bf84a454d49b6497c16ed15c84a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.720448 5412 tablet_service.cc:1514] Processing DeleteTablet for tablet e7eef600626a4961ab1fae27162537bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.720671 5720 raft_consensus.cc:2454] T 7d89e4185229431f9a68f599da4aff34 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:17.721004 5377 leader_election.cc:302] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.721261 7552 raft_consensus.cc:2781] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.721423 7552 raft_consensus.cc:686] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:17.722083 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19e8ef286cf247278fe2b01218e6310a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.722364 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c2d87c949a4bd18e0541bad7606e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.725211 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3be7b44034d25ab402f11166e2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.725775 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05751530548b4738b462e9c3cd9f0dea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.721681 7552 consensus_queue.cc:237] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.728888 5412 tablet_service.cc:1467] Processing CreateTablet for tablet e777995062df40b28007f4e0e2baea81 (DEFAULT_TABLE table=default.Table_3004daed58924ac899ae89618d420197 [id=5e553ff7abab4d74b541d98546545c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.729214 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e777995062df40b28007f4e0e2baea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.729869 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c1a5919b5247c8aa542b806611dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.729916 5697 tablet_service.cc:1467] Processing CreateTablet for tablet e777995062df40b28007f4e0e2baea81 (DEFAULT_TABLE table=default.Table_3004daed58924ac899ae89618d420197 [id=5e553ff7abab4d74b541d98546545c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.730125 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9354ceaf6f6d4f529d9dbe7b78360b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.730371 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e777995062df40b28007f4e0e2baea81. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:17.733399 5656 consensus_peers.cc:485] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53bc9bf1c04e4015a152cb750184c9ce. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.733873 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00091474fd5c487d96db0d3d61665b81. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:17.735072 5654 consensus_peers.cc:485] T 9211028e9eeb452d8ab5903710349ca6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:17.736003 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f1cc86864eaa4b64a5b7de5c559b25a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.736539 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1cc86864eaa4b64a5b7de5c559b25a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.737890 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 400e3f14d7764cc7b36d79037838fb9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.738075 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3859684aac9b4792975cbc22c95a6775. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.738528 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_14a9319948f348828122c47d5271ab38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.738967 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14a9319948f348828122c47d5271ab38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.740417 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 2692e4af55174d42bf0035f52f004992 (DEFAULT_TABLE table=default.Table_50f7ec043384417799974a1e441c4747 [id=b23662d6cc5c4af59a2ca2007af661f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.740725 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2692e4af55174d42bf0035f52f004992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.742754 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_84b116dd65724393b389f4455634ac10" } new_table_name: "default.Table_04721347f96042e5ac28d63b0a75a9fa" modify_external_catalogs: true
I20240507 08:12:17.743432 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3be7b44034d25ab402f11166e2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.745774 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.745990 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0add560e10e04e7c8507e0d8c4632dab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.749295 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6a769d7d87b84f1da5328510e326530f" } new_table_name: "default.Table_0253d1dd03a9452085ea72b226b4278f" modify_external_catalogs: true
I20240507 08:12:17.749668 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1c1a5919b5247c8aa542b806611dfe7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.749856 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63c2d87c949a4bd18e0541bad7606e02. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.751284 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5170fb1c78db4149a6ee865b227a5f6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.753782 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc3d1e9e7e53483fa75f7cd918295e50. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.756080 7358 raft_consensus.cc:482] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.756212 7358 raft_consensus.cc:504] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.756772 7358 leader_election.cc:288] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.757616 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd3dbd84234e398a6151483d447260" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.757813 5445 raft_consensus.cc:2454] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:17.758518 5656 leader_election.cc:302] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:17.758944 7100 raft_consensus.cc:2781] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:17.759037 7100 raft_consensus.cc:482] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:17.759112 7100 raft_consensus.cc:3037] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.759461 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 75b716bdf47941049a6e247b4518019e (DEFAULT_TABLE table=default.Table_d3a39ab037194d1b9c28f520890fa695 [id=aa8eafe821da45d6b767d47d765698a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.759799 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b716bdf47941049a6e247b4518019e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.761399 7100 raft_consensus.cc:504] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.766177 7100 leader_election.cc:288] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:17.770190 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 7415b8bff791418db662f0734a7e352a (DEFAULT_TABLE table=default.Table_881768862fbe460d961c048ed94bd724 [id=4c9ebe5dc3c3406f9a743208eef3820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.770582 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7415b8bff791418db662f0734a7e352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.770504 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_830c53deefa541df8e6682e8c3ef976f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.770802 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1 (DEFAULT_TABLE table=default.Table_47429eec1f5942329f871ca8dd068b20 [id=edcf4fc63afe4892a8b77b52234df4a4]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.771011 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_830c53deefa541df8e6682e8c3ef976f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.771091 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.772140 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d54c9b59e4684e0482f46ffb7801ee02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.772560 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d54c9b59e4684e0482f46ffb7801ee02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.770190 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 88f33d4681fa4b42bed556412240c355 (DEFAULT_TABLE table=default.Table_1acd7fcd5bbd40d3ad61d63adf97169e [id=6fea074c2aa24910ab044cc691eca3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.773545 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5a685699b57943ee8abbfa913c98aa95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.773873 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f33d4681fa4b42bed556412240c355. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:17.773942 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a685699b57943ee8abbfa913c98aa95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.775987 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1140) took 3430 ms. Trace:
I20240507 08:12:17.775867 7550 consensus_queue.cc:1035] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:17.776055 6513 rpcz_store.cc:276] 0507 08:12:14.345123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.345165 (+ 42us) service_pool.cc:225] Handling call
0507 08:12:17.563435 (+3218270us) ts_tablet_manager.cc:1120] Deleting tablet 85dc58133d2b4f5496b4e570ae6494c0
0507 08:12:17.563445 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.581835 (+ 18390us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.584681 (+ 2846us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.775973 (+191292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3218132}
I20240507 08:12:17.776310 5840 tablet_bootstrap.cc:492] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:17.776644 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 (DEFAULT_TABLE table=default.Table_9bdc371ba30a4c879ec337f2a1d3e90e [id=0212839f6e604fbaaf4bcb47afb71203]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.776971 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd0f775bfa344b9b02afc875db46af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.776932 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a63ec02af6ab4e85830582d44267f8c1" } new_table_name: "default.Table_e3bcf078654d42ed8331d033b29e482c" modify_external_catalogs: true
I20240507 08:12:17.777757 5840 tablet_bootstrap.cc:654] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.777791 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 94eafbab0c304f24abe35d53dcfd6919 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.778066 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eafbab0c304f24abe35d53dcfd6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.776332 6513 tablet_replica.cc:330] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:17.779060 6513 raft_consensus.cc:2229] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.779289 6513 raft_consensus.cc:2258] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.779827 6513 ts_tablet_manager.cc:1875] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.781730 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 85dc58133d2b4f5496b4e570ae6494c0 (table default.Table_f7a84b84863d462192343bcb92bd888d [id=5714d419aa014285ac6779ef49cc29fa]) successfully deleted
I20240507 08:12:17.789094 5231 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069530908792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:12:17.789315 5231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20240507 08:12:17.791067 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd3dbd84234e398a6151483d447260" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:17.791270 5445 raft_consensus.cc:3037] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:17.720968 5517 leader_election.cc:341] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 31251576beee4a8a81a453b721c9c058
I20240507 08:12:17.774991 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd3dbd84234e398a6151483d447260" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:17.795244 5517 leader_election.cc:302] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.759414 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66cd3dbd84234e398a6151483d447260" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:17.795579 5517 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 74619us
W20240507 08:12:17.795637 5654 leader_election.cc:341] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:17.795785 5517 leader_election.cc:341] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 565ac6091f5e48b1b024094d5560e275
W20240507 08:12:17.795816 5654 leader_election.cc:341] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.795873 5517 leader_election.cc:302] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.796141 7543 raft_consensus.cc:2726] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.796509 7543 raft_consensus.cc:2726] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.730181 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 54e3be7b44034d25ab402f11166e2ab5 (DEFAULT_TABLE table=default.Table_7ef0ec9add454eb28c8bb9725ee5274f [id=e2ed23c303c34e25942822f70772c7ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.798499 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1162) took 2460 ms. Trace:
I20240507 08:12:17.798650 5841 rpcz_store.cc:276] 0507 08:12:15.337918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.338157 (+ 239us) service_pool.cc:225] Handling call
0507 08:12:17.608785 (+2270628us) ts_tablet_manager.cc:1120] Deleting tablet 8a015bf84a454d49b6497c16ed15c84a
0507 08:12:17.608794 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.615936 (+ 7142us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.625341 (+ 9405us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.798476 (+173135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2258996}
I20240507 08:12:17.799113 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1163) took 2461 ms. Trace:
I20240507 08:12:17.799244 5841 rpcz_store.cc:276] 0507 08:12:15.338011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.338338 (+ 327us) service_pool.cc:225] Handling call
0507 08:12:17.799020 (+2460682us) ts_tablet_manager.cc:1120] Deleting tablet 957775c01be94cc081db344c95a81310
0507 08:12:17.799030 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.799097 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2448004}
I20240507 08:12:17.799568 5841 tablet_replica.cc:330] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:17.799822 5841 raft_consensus.cc:2229] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.800056 5841 raft_consensus.cc:2258] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.801164 5841 ts_tablet_manager.cc:1875] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.805094 7079 ts_tablet_manager.cc:1888] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:17.805670 7079 log.cc:1196] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/0d617213b959452ab6519809e56db128
I20240507 08:12:17.806007 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12a6149f27754dcfa4b02b2a1a53b6e5" } modify_external_catalogs: true
I20240507 08:12:17.806217 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_12a6149f27754dcfa4b02b2a1a53b6e5" } modify_external_catalogs: true
I20240507 08:12:17.807307 7079 ts_tablet_manager.cc:1909] T 0d617213b959452ab6519809e56db128 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:17.718308 7590 raft_consensus.cc:504] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.808176 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1133) took 2118 ms. Trace:
I20240507 08:12:17.808748 7590 leader_election.cc:288] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:17.811859 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd97542e3db0413c8272ff2971801be7 consensus state has no leader
I20240507 08:12:17.815140 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:17.818152 5264 catalog_manager.cc:5484] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:17.808645 7079 rpcz_store.cc:276] 0507 08:12:15.689967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.690103 (+ 136us) service_pool.cc:225] Handling call
0507 08:12:17.696884 (+2006781us) ts_tablet_manager.cc:1120] Deleting tablet 0d617213b959452ab6519809e56db128
0507 08:12:17.696895 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.716396 (+ 19501us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.804975 (+ 88579us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.808154 (+ 3179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006635}
I20240507 08:12:17.819046 7079 tablet_replica.cc:330] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.818807 5264 catalog_manager.cc:5484] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:17.819286 7079 raft_consensus.cc:2229] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.819433 7079 raft_consensus.cc:2258] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.819922 7079 ts_tablet_manager.cc:1875] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.809803 5839 tablet_bootstrap.cc:492] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.820150 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 0d617213b959452ab6519809e56db128 (table default.Table_9b12ea39074c4f9c8796a9fae7c8e5a6 [id=bce0bd66e7b74db7934937410c81b20b]) successfully deleted
I20240507 08:12:17.820312 5839 ts_tablet_manager.cc:1367] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.301s user 0.004s sys 0.002s
I20240507 08:12:17.813758 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 2692e4af55174d42bf0035f52f004992 (DEFAULT_TABLE table=default.Table_50f7ec043384417799974a1e441c4747 [id=b23662d6cc5c4af59a2ca2007af661f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.814180 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1 (DEFAULT_TABLE table=default.Table_47429eec1f5942329f871ca8dd068b20 [id=edcf4fc63afe4892a8b77b52234df4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.814781 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 94eafbab0c304f24abe35d53dcfd6919 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.815037 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 88f33d4681fa4b42bed556412240c355 (DEFAULT_TABLE table=default.Table_1acd7fcd5bbd40d3ad61d63adf97169e [id=6fea074c2aa24910ab044cc691eca3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.815348 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 75b716bdf47941049a6e247b4518019e (DEFAULT_TABLE table=default.Table_d3a39ab037194d1b9c28f520890fa695 [id=aa8eafe821da45d6b767d47d765698a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.820956 5839 raft_consensus.cc:348] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.821131 5839 raft_consensus.cc:374] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.815863 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 7415b8bff791418db662f0734a7e352a (DEFAULT_TABLE table=default.Table_881768862fbe460d961c048ed94bd724 [id=4c9ebe5dc3c3406f9a743208eef3820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.821187 5839 raft_consensus.cc:729] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.816277 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 (DEFAULT_TABLE table=default.Table_9bdc371ba30a4c879ec337f2a1d3e90e [id=0212839f6e604fbaaf4bcb47afb71203]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.821383 5839 consensus_queue.cc:260] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.826653 5839 ts_tablet_manager.cc:1398] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
W20240507 08:12:17.826921 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d1a4e05273b4f8c83dc8c7825a39310 consensus state has no leader
I20240507 08:12:17.828004 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54e3be7b44034d25ab402f11166e2ab5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.828351 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd0f775bfa344b9b02afc875db46af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.829268 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eafbab0c304f24abe35d53dcfd6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.829762 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7415b8bff791418db662f0734a7e352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.834184 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b716bdf47941049a6e247b4518019e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.834509 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2692e4af55174d42bf0035f52f004992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.837704 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f33d4681fa4b42bed556412240c355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.838455 5839 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.840296 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.840900 5839 tablet_bootstrap.cc:654] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.845794 7079 ts_tablet_manager.cc:1888] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:17.850389 7079 log.cc:1196] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b0b9f85cd78748d3afce83b64b3f823b
I20240507 08:12:17.850793 7079 ts_tablet_manager.cc:1909] T b0b9f85cd78748d3afce83b64b3f823b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:17.851161 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1134) took 2161 ms. Trace:
I20240507 08:12:17.851367 7079 rpcz_store.cc:276] 0507 08:12:15.690058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.690107 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:17.819002 (+2128895us) ts_tablet_manager.cc:1120] Deleting tablet b0b9f85cd78748d3afce83b64b3f823b
0507 08:12:17.819011 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.835880 (+ 16869us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.845719 (+ 9839us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.851145 (+ 5426us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2128642}
I20240507 08:12:17.851828 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b0b9f85cd78748d3afce83b64b3f823b (table default.Table_4a0c1ff46db94e2791a9c86887b20998 [id=8752e02495334c148e42bea97bf052b3]) successfully deleted
I20240507 08:12:17.852834 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e8d09d71d5ff4cd68efa2496e75c47d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.853168 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1132) took 2163 ms. Trace:
I20240507 08:12:17.853235 7079 rpcz_store.cc:276] 0507 08:12:15.689846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.690431 (+ 585us) service_pool.cc:225] Handling call
0507 08:12:17.853083 (+2162652us) ts_tablet_manager.cc:1120] Deleting tablet 1037c3c0af5e464bb23a311e552fcf21
0507 08:12:17.853092 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.853153 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2162522}
W20240507 08:12:17.853359 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8d09d71d5ff4cd68efa2496e75c47d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:17.853538 5249 catalog_manager.cc:4797] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 1037c3c0af5e464bb23a311e552fcf21 because the tablet was not found. No further retry: Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21
I20240507 08:12:17.854293 5555 tablet_service.cc:1514] Processing DeleteTablet for tablet a56cb45381604ac9a69e6f3edaa4eefc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.854674 5561 tablet_service.cc:1467] Processing CreateTablet for tablet e777995062df40b28007f4e0e2baea81 (DEFAULT_TABLE table=default.Table_3004daed58924ac899ae89618d420197 [id=5e553ff7abab4d74b541d98546545c6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.854907 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_99c89f1c9f494e4786dba165357d1b31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.855345 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99c89f1c9f494e4786dba165357d1b31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.856035 7079 tablet_replica.cc:330] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:17.856197 7079 raft_consensus.cc:2229] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:17.856320 7079 raft_consensus.cc:2258] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:17.856498 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5abbb74539645c2bebcc549a43cd836" } modify_external_catalogs: true
I20240507 08:12:17.856658 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5abbb74539645c2bebcc549a43cd836" } modify_external_catalogs: true
I20240507 08:12:17.857416 7079 ts_tablet_manager.cc:1875] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:17.857738 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0253d1dd03a9452085ea72b226b4278f" } modify_external_catalogs: true
I20240507 08:12:17.857864 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0253d1dd03a9452085ea72b226b4278f" } modify_external_catalogs: true
I20240507 08:12:17.859287 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_38e25aba16ae4bcab73abbb11562bfd9" } new_table_name: "default.Table_2c4d993ae928415181f53571b6bf0815" modify_external_catalogs: true
I20240507 08:12:17.860563 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0dba2acabcc144e5929e28d0332ed1df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.861111 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0dba2acabcc144e5929e28d0332ed1df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.862429 5840 tablet_bootstrap.cc:492] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:17.863394 5444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb1901ddec07495abba085e7188c00d0" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:17.864522 5445 raft_consensus.cc:2454] T 66cd3dbd84234e398a6151483d447260 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:17.865694 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e777995062df40b28007f4e0e2baea81. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.866281 5520 leader_election.cc:302] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.866525 7590 raft_consensus.cc:2726] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.874231 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6cc465fb947d406f85b3779471ec43cc" } modify_external_catalogs: true
I20240507 08:12:17.874429 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6cc465fb947d406f85b3779471ec43cc" } modify_external_catalogs: true
W20240507 08:12:17.876765 5378 consensus_peers.cc:485] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.877418 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 2692e4af55174d42bf0035f52f004992 (DEFAULT_TABLE table=default.Table_50f7ec043384417799974a1e441c4747 [id=b23662d6cc5c4af59a2ca2007af661f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.878403 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8a015bf84a454d49b6497c16ed15c84a (table default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]) successfully deleted
W20240507 08:12:17.878616 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 957775c01be94cc081db344c95a81310 because the tablet was not found. No further retry: Not found: Tablet not found: 957775c01be94cc081db344c95a81310
I20240507 08:12:17.879537 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2692e4af55174d42bf0035f52f004992. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.879020 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1 (DEFAULT_TABLE table=default.Table_47429eec1f5942329f871ca8dd068b20 [id=edcf4fc63afe4892a8b77b52234df4a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.880885 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 88f33d4681fa4b42bed556412240c355 (DEFAULT_TABLE table=default.Table_1acd7fcd5bbd40d3ad61d63adf97169e [id=6fea074c2aa24910ab044cc691eca3b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.882176 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 75b716bdf47941049a6e247b4518019e (DEFAULT_TABLE table=default.Table_d3a39ab037194d1b9c28f520890fa695 [id=aa8eafe821da45d6b767d47d765698a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.882802 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 94eafbab0c304f24abe35d53dcfd6919 (DEFAULT_TABLE table=default.Table_2472cc2477394ac9addcebfd33bea4aa [id=23a9ae66a7f446d2abb3d5cb61117ec9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.883111 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 with delete_type TABLET_DATA_DELETED (replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.883296 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 (DEFAULT_TABLE table=default.Table_9bdc371ba30a4c879ec337f2a1d3e90e [id=0212839f6e604fbaaf4bcb47afb71203]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.883562 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 7415b8bff791418db662f0734a7e352a (DEFAULT_TABLE table=default.Table_881768862fbe460d961c048ed94bd724 [id=4c9ebe5dc3c3406f9a743208eef3820c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.883754 5843 tablet_bootstrap.cc:654] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.883802 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 with delete_type TABLET_DATA_DELETED (replaced by tablet f89efb8414dc49369d361adf908927e6) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.889791 5582 raft_consensus.cc:3037] T eb1901ddec07495abba085e7188c00d0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:17.894470 7550 raft_consensus.cc:482] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.894717 7550 raft_consensus.cc:504] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:17.897801 7550 leader_election.cc:288] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:17.898335 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:17.898912 5378 leader_election.cc:341] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.901793 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94eafbab0c304f24abe35d53dcfd6919. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.902267 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bd0f775bfa344b9b02afc875db46af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.902738 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88f33d4681fa4b42bed556412240c355. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.903180 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75b716bdf47941049a6e247b4518019e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.904450 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7415b8bff791418db662f0734a7e352a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.904879 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b114f0a8da84a0186d8ed4e0c89d9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.914053 5841 ts_tablet_manager.cc:1888] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:17.914172 5841 log.cc:1196] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c707458d1c094d87b2204683b28b49e4
I20240507 08:12:17.914535 5841 ts_tablet_manager.cc:1909] T c707458d1c094d87b2204683b28b49e4 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:17.865869 5656 leader_election.cc:302] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.919265 5840 ts_tablet_manager.cc:1367] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.143s user 0.007s sys 0.000s
W20240507 08:12:17.919428 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 53587us
I20240507 08:12:17.919816 7100 raft_consensus.cc:2781] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:17.919945 7100 raft_consensus.cc:686] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:17.920027 5840 raft_consensus.cc:348] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.920236 5840 raft_consensus.cc:374] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.920181 7100 consensus_queue.cc:237] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.920301 5840 raft_consensus.cc:729] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.897935 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:17.920616 5840 consensus_queue.cc:260] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:17.920924 5377 leader_election.cc:341] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:17.921027 5377 leader_election.cc:302] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:17.921147 5840 ts_tablet_manager.cc:1398] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:17.925551 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 059300a5cbc440db90e80a302d6e6518 (DEFAULT_TABLE table=default.Table_428bd11419b24adf8a02eb81b03d995f [id=1d44ad4af247439186f55ef0fc3a2f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.926013 5414 tablet_service.cc:1467] Processing CreateTablet for tablet d6e322ea08f9401fa035e0a1861b91af (DEFAULT_TABLE table=default.Table_eb9840182697421db11214ad83ad8b24 [id=57f39cf3bf874103957def1d82354917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.926270 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059300a5cbc440db90e80a302d6e6518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.928486 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 92d34e7394dc41398be91943a258ae7e (DEFAULT_TABLE table=default.Table_0a23d6a405694a7582422d77797c40ae [id=df0d84c1319c4ff6a782ac38f85d2850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.929491 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 059300a5cbc440db90e80a302d6e6518 (DEFAULT_TABLE table=default.Table_428bd11419b24adf8a02eb81b03d995f [id=1d44ad4af247439186f55ef0fc3a2f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.929584 5685 tablet_service.cc:1467] Processing CreateTablet for tablet d6e322ea08f9401fa035e0a1861b91af (DEFAULT_TABLE table=default.Table_eb9840182697421db11214ad83ad8b24 [id=57f39cf3bf874103957def1d82354917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.929615 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 92d34e7394dc41398be91943a258ae7e (DEFAULT_TABLE table=default.Table_0a23d6a405694a7582422d77797c40ae [id=df0d84c1319c4ff6a782ac38f85d2850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.930848 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 92d34e7394dc41398be91943a258ae7e (DEFAULT_TABLE table=default.Table_0a23d6a405694a7582422d77797c40ae [id=df0d84c1319c4ff6a782ac38f85d2850]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.930990 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d6e322ea08f9401fa035e0a1861b91af (DEFAULT_TABLE table=default.Table_eb9840182697421db11214ad83ad8b24 [id=57f39cf3bf874103957def1d82354917]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.931326 7378 raft_consensus.cc:2726] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.931835 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d34e7394dc41398be91943a258ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.932688 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fd204b1d0b0943bcbf82f683d5a71256" } modify_external_catalogs: true
I20240507 08:12:17.932850 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fd204b1d0b0943bcbf82f683d5a71256" } modify_external_catalogs: true
I20240507 08:12:17.934908 5839 tablet_bootstrap.cc:492] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:17.935020 5839 ts_tablet_manager.cc:1367] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.097s user 0.005s sys 0.000s
I20240507 08:12:17.936952 5420 tablet_service.cc:1467] Processing CreateTablet for tablet ac28170d9c3648b79a4394c7e51334bf (DEFAULT_TABLE table=default.Table_5b9ad4cb5d684e6197ffffffe75bc7ed [id=d212622018874e3ebde08cdcf344d57f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:17.942407 5654 consensus_peers.cc:485] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c707458d1c094d87b2204683b28b49e4. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.941243 5839 raft_consensus.cc:348] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.942879 5839 raft_consensus.cc:374] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.943702 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:17.943467 5839 raft_consensus.cc:729] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.948093 5558 tablet_service.cc:1467] Processing CreateTablet for tablet ac28170d9c3648b79a4394c7e51334bf (DEFAULT_TABLE table=default.Table_5b9ad4cb5d684e6197ffffffe75bc7ed [id=d212622018874e3ebde08cdcf344d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.949071 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28170d9c3648b79a4394c7e51334bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.950349 6513 ts_tablet_manager.cc:1888] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20240507 08:12:17.950399 5654 consensus_peers.cc:485] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.950640 6513 log.cc:1196] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b0542325c80241fca7e317cda64302a9
I20240507 08:12:17.950625 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 059300a5cbc440db90e80a302d6e6518 (DEFAULT_TABLE table=default.Table_428bd11419b24adf8a02eb81b03d995f [id=1d44ad4af247439186f55ef0fc3a2f2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.951408 6513 ts_tablet_manager.cc:1909] T b0542325c80241fca7e317cda64302a9 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:17.953590 5656 consensus_peers.cc:485] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f01e3311a8b4553a31914922fb08d13. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:17.953867 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e322ea08f9401fa035e0a1861b91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.954393 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059300a5cbc440db90e80a302d6e6518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:17.954520 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d718aa8fe99041688964d1a05cef49a0
I20240507 08:12:17.958245 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet d718aa8fe99041688964d1a05cef49a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.959100 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet d718aa8fe99041688964d1a05cef49a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:17.960831 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet d718aa8fe99041688964d1a05cef49a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:17.962913 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_16a1c2b0c7bd4504a1d2c6b472ccecbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:17.963454 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16a1c2b0c7bd4504a1d2c6b472ccecbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.964463 5843 tablet_bootstrap.cc:492] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:17.964638 5843 ts_tablet_manager.cc:1367] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.323s user 0.003s sys 0.002s
I20240507 08:12:17.965812 5843 raft_consensus.cc:348] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.971483 5843 raft_consensus.cc:374] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:17.971611 5843 raft_consensus.cc:729] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:17.973786 5843 consensus_queue.cc:260] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.974702 7590 raft_consensus.cc:482] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:17.974828 7590 raft_consensus.cc:504] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.975483 7590 leader_election.cc:288] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:17.977056 5520 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 34adee00b529424b9f274c61ce73b0bc
W20240507 08:12:17.977192 5517 leader_election.cc:341] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 34adee00b529424b9f274c61ce73b0bc
I20240507 08:12:17.977321 5517 leader_election.cc:302] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:17.977275 5839 consensus_queue.cc:260] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:17.977739 7590 raft_consensus.cc:2726] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:17.977903 5839 ts_tablet_manager.cc:1398] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20240507 08:12:17.978164 5839 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:17.979717 5839 tablet_bootstrap.cc:654] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:17.980594 5843 ts_tablet_manager.cc:1398] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20240507 08:12:17.983500 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_edb3317ccbd440c99457c0d12ce3c25c" } new_table_name: "default.Table_19c6faaf4dcb41f9be8ec2ba7e00794b" modify_external_catalogs: true
I20240507 08:12:17.986171 5843 tablet_bootstrap.cc:492] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:17.991840 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1bc794eff3b343839470ca0506214d11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:17.992774 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:17.992921 7100 consensus_queue.cc:1035] T 7db7b8c560f048f4b02febd371aa2a8c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240507 08:12:17.993680 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bc794eff3b343839470ca0506214d11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:17.997565 5683 tablet_service.cc:1467] Processing CreateTablet for tablet ac28170d9c3648b79a4394c7e51334bf (DEFAULT_TABLE table=default.Table_5b9ad4cb5d684e6197ffffffe75bc7ed [id=d212622018874e3ebde08cdcf344d57f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.998746 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 11feca1cf7924e08958a538b96883c21 (DEFAULT_TABLE table=default.Table_a00f64d4fd4146a4bad06d8aa738d872 [id=ae0430787a25415da564030210943d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.999089 5548 tablet_service.cc:1467] Processing CreateTablet for tablet ba5d52927ec748c6bc543ccb9c95cce6 (DEFAULT_TABLE table=default.Table_ec4d03ca44c54019831f558891aaa6e3 [id=481364cab40d4053870812a266077240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.999716 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 11feca1cf7924e08958a538b96883c21 (DEFAULT_TABLE table=default.Table_a00f64d4fd4146a4bad06d8aa738d872 [id=ae0430787a25415da564030210943d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.000936 5421 tablet_service.cc:1467] Processing CreateTablet for tablet ba5d52927ec748c6bc543ccb9c95cce6 (DEFAULT_TABLE table=default.Table_ec4d03ca44c54019831f558891aaa6e3 [id=481364cab40d4053870812a266077240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:17.975306 7613 tablet.cc:1722] T bcefbe3a60f84cadb5fd3e69c6efdd02 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:18.002177 5689 tablet_service.cc:1467] Processing CreateTablet for tablet ba5d52927ec748c6bc543ccb9c95cce6 (DEFAULT_TABLE table=default.Table_ec4d03ca44c54019831f558891aaa6e3 [id=481364cab40d4053870812a266077240]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.005738 7610 tablet.cc:1722] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:18.006443 7612 tablet.cc:1722] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:18.010587 7611 tablet.cc:1722] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:18.013909 7079 ts_tablet_manager.cc:1888] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.014225 7079 log.cc:1196] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e298f11ff9f54e48b8de435f6849f39a
I20240507 08:12:18.015070 7079 ts_tablet_manager.cc:1909] T e298f11ff9f54e48b8de435f6849f39a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:18.016717 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 85b9b4b5d2884fc993cfae9b41824146 (DEFAULT_TABLE table=default.Table_3c8c77f7212e495eb588a34553d6f7d9 [id=188913b9fda74702aa224e2a643515d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.016700 5262 catalog_manager.cc:5484] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:18.019474 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 85b9b4b5d2884fc993cfae9b41824146 (DEFAULT_TABLE table=default.Table_3c8c77f7212e495eb588a34553d6f7d9 [id=188913b9fda74702aa224e2a643515d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.022816 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 85b9b4b5d2884fc993cfae9b41824146 (DEFAULT_TABLE table=default.Table_3c8c77f7212e495eb588a34553d6f7d9 [id=188913b9fda74702aa224e2a643515d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.025765 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11feca1cf7924e08958a538b96883c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.026029 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28170d9c3648b79a4394c7e51334bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.026316 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b9b4b5d2884fc993cfae9b41824146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.026535 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d34e7394dc41398be91943a258ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.026850 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5d52927ec748c6bc543ccb9c95cce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.027143 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e322ea08f9401fa035e0a1861b91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.027465 7358 raft_consensus.cc:482] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.027582 7358 raft_consensus.cc:504] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.028168 7358 leader_election.cc:288] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.028712 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.028887 5442 raft_consensus.cc:2454] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:18.029209 5656 leader_election.cc:302] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.029431 7394 raft_consensus.cc:2781] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:18.029490 7394 raft_consensus.cc:482] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:18.029546 7394 raft_consensus.cc:3037] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.030405 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cd4d1fa6a8d94af982e1d23c08cbdeac
I20240507 08:12:18.030543 5843 tablet_bootstrap.cc:654] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.032047 7394 raft_consensus.cc:504] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.032608 7394 leader_election.cc:288] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.032793 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.033713 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:18.033870 5442 raft_consensus.cc:3037] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.034804 5565 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.036197 5416 tablet_service.cc:1467] Processing CreateTablet for tablet e766383f2adc47e1affa46ec627d652a (DEFAULT_TABLE table=default.Table_c13e53ccc0144cb4a2fd3cce833674f6 [id=7aca3bcc7c5245adbe523b9d878e1214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.036361 5442 raft_consensus.cc:2454] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:18.036343 5563 tablet_service.cc:1467] Processing CreateTablet for tablet e766383f2adc47e1affa46ec627d652a (DEFAULT_TABLE table=default.Table_c13e53ccc0144cb4a2fd3cce833674f6 [id=7aca3bcc7c5245adbe523b9d878e1214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.036515 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4d1fa6a8d94af982e1d23c08cbdeac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.039098 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:18.040827 5656 leader_election.cc:302] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.041111 7358 raft_consensus.cc:2781] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:18.041244 7358 raft_consensus.cc:686] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:18.041489 7358 consensus_queue.cc:237] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.045343 5843 tablet_bootstrap.cc:492] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:18.045683 5843 ts_tablet_manager.cc:1367] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.060s user 0.002s sys 0.002s
I20240507 08:12:18.047031 5843 raft_consensus.cc:348] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.047788 5843 raft_consensus.cc:374] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.047981 5843 raft_consensus.cc:729] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.048334 5843 consensus_queue.cc:260] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.049345 5843 ts_tablet_manager.cc:1398] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:18.049997 5843 tablet_bootstrap.cc:492] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:18.051505 5843 tablet_bootstrap.cc:654] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:18.033833 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 449d6c8270184ce2918cee8ce731f788 consensus state has no leader
I20240507 08:12:18.037802 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8d1a4e05273b4f8c83dc8c7825a39310
I20240507 08:12:18.054208 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_775645191cbe4f3aafd1b0ee37db5591" } new_table_name: "default.Table_7864801aee33473594610fd24e6ae111" modify_external_catalogs: true
I20240507 08:12:18.033550 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_165fc764f58349838a6022d9320583ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.055708 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_165fc764f58349838a6022d9320583ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:18.059442 5656 consensus_peers.cc:485] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c707458d1c094d87b2204683b28b49e4. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:18.059883 5654 consensus_peers.cc:485] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.059652 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4ae0974a73994e749cf5734401312e03" } new_table_name: "default.Table_d384a5d7c29a4df492a4cfaae5d18044" modify_external_catalogs: true
I20240507 08:12:18.062247 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.062546 5690 tablet_service.cc:1467] Processing CreateTablet for tablet e766383f2adc47e1affa46ec627d652a (DEFAULT_TABLE table=default.Table_c13e53ccc0144cb4a2fd3cce833674f6 [id=7aca3bcc7c5245adbe523b9d878e1214]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.058691 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.059917 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet b60ec15a83934999871425ef9a64ec72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.063617 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e766383f2adc47e1affa46ec627d652a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.067332 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf0c092cf64d2786382d5e9c2b2e72 (DEFAULT_TABLE table=default.Table_3d79cddfa377454d99a02c32c919b6a5 [id=e5d732adf0834bd69e3e9b1d0c9fe408]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.047224 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_817b15f93fff45d69d52c60a2300e671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:18.067740 5692 tablet_service.cc:1467] Processing CreateTablet for tablet d0eed609cb24474592c5b66b63e7669f (DEFAULT_TABLE table=default.Table_454ae1a1af424a77a79ea0010b86a0bb [id=d191061a79464d52afb0198638bdf7ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:18.067868 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_817b15f93fff45d69d52c60a2300e671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.067023 5410 tablet_service.cc:1467] Processing CreateTablet for tablet d0eed609cb24474592c5b66b63e7669f (DEFAULT_TABLE table=default.Table_454ae1a1af424a77a79ea0010b86a0bb [id=d191061a79464d52afb0198638bdf7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.028474 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "167ee9cbc9174b82824c468e098472a6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:18.076661 5654 leader_election.cc:341] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.076797 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5d52927ec748c6bc543ccb9c95cce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.077345 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 059300a5cbc440db90e80a302d6e6518. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.077489 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1135) took 2381 ms. Trace:
I20240507 08:12:18.077780 6675 consensus_queue.cc:1035] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.078806 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c830acb24c5c899eecff2c6b8bff (DEFAULT_TABLE table=default.Table_edb3317ccbd440c99457c0d12ce3c25c [id=1f500681aaac43a7b13b2c4a369e7bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.079138 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c830acb24c5c899eecff2c6b8bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.079311 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6e322ea08f9401fa035e0a1861b91af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.079957 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 192492075dbf4badb8c70a9546ddd80a (DEFAULT_TABLE table=default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.080252 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192492075dbf4badb8c70a9546ddd80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.081809 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92d34e7394dc41398be91943a258ae7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.081686 7079 rpcz_store.cc:276] 0507 08:12:15.695540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.695584 (+ 44us) service_pool.cc:225] Handling call
0507 08:12:17.855997 (+2160413us) ts_tablet_manager.cc:1120] Deleting tablet e298f11ff9f54e48b8de435f6849f39a
0507 08:12:17.856006 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.885208 (+ 29202us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.013699 (+128491us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.077469 (+ 63770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2160273}
I20240507 08:12:18.082065 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1136) took 2358 ms. Trace:
I20240507 08:12:18.082134 7079 rpcz_store.cc:276] 0507 08:12:15.723218 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.723268 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:18.081949 (+2358681us) ts_tablet_manager.cc:1120] Deleting tablet ddcb720c09c5486995391837fc54e7ce
0507 08:12:18.081961 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.082049 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2358498}
I20240507 08:12:18.082401 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1137) took 2355 ms. Trace:
I20240507 08:12:18.082453 7079 rpcz_store.cc:276] 0507 08:12:15.727058 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.727113 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:18.082325 (+2355212us) ts_tablet_manager.cc:1120] Deleting tablet f9b60e7fd01c4c96911fff6bf62ade96
0507 08:12:18.082331 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.082390 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2355034}
I20240507 08:12:18.077725 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b9b4b5d2884fc993cfae9b41824146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.086642 5693 tablet_service.cc:1467] Processing CreateTablet for tablet f859146c376c4b1eb52bff953f05837b (DEFAULT_TABLE table=default.Table_6cc465fb947d406f85b3779471ec43cc [id=81c7c926583347188ad733e695b06d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.087021 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f859146c376c4b1eb52bff953f05837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.087188 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6d5dc177474ca32f9a3154a59e92 (DEFAULT_TABLE table=default.Table_8d8d78277b8044a7b75cc3409a82ef21 [id=f7e6ac0a045d4b2f91debde6fa3be7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.087518 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6d5dc177474ca32f9a3154a59e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.090063 5699 tablet_service.cc:1467] Processing CreateTablet for tablet c4b78ec5ea7c43149c89be8dd2df0e45 (DEFAULT_TABLE table=default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6 [id=6d327ba6d58a4d2caed8b602db929fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.090349 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b78ec5ea7c43149c89be8dd2df0e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.092291 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 192492075dbf4badb8c70a9546ddd80a (DEFAULT_TABLE table=default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.092621 5409 tablet_service.cc:1467] Processing CreateTablet for tablet c4b78ec5ea7c43149c89be8dd2df0e45 (DEFAULT_TABLE table=default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6 [id=6d327ba6d58a4d2caed8b602db929fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.092660 5417 tablet_service.cc:1467] Processing CreateTablet for tablet f859146c376c4b1eb52bff953f05837b (DEFAULT_TABLE table=default.Table_6cc465fb947d406f85b3779471ec43cc [id=81c7c926583347188ad733e695b06d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.092955 5422 tablet_service.cc:1467] Processing CreateTablet for tablet decb2294f6cd4800b332eccab802d224 (DEFAULT_TABLE table=default.Table_7ab1a575a68349d4a23fb56a2fd5098b [id=ab2c9bf1437b454f8d30a179d61fdea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.092630 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6d5dc177474ca32f9a3154a59e92 (DEFAULT_TABLE table=default.Table_8d8d78277b8044a7b75cc3409a82ef21 [id=f7e6ac0a045d4b2f91debde6fa3be7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.093967 5684 tablet_service.cc:1467] Processing CreateTablet for tablet decb2294f6cd4800b332eccab802d224 (DEFAULT_TABLE table=default.Table_7ab1a575a68349d4a23fb56a2fd5098b [id=ab2c9bf1437b454f8d30a179d61fdea7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.094288 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decb2294f6cd4800b332eccab802d224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.069090 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf0c092cf64d2786382d5e9c2b2e72 (DEFAULT_TABLE table=default.Table_3d79cddfa377454d99a02c32c919b6a5 [id=e5d732adf0834bd69e3e9b1d0c9fe408]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.073014 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c830acb24c5c899eecff2c6b8bff (DEFAULT_TABLE table=default.Table_edb3317ccbd440c99457c0d12ce3c25c [id=1f500681aaac43a7b13b2c4a369e7bd7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:18.095422 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
I20240507 08:12:18.098446 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1141) took 3741 ms. Trace:
I20240507 08:12:18.100770 6513 rpcz_store.cc:276] 0507 08:12:14.356550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.356600 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:17.776297 (+3419697us) ts_tablet_manager.cc:1120] Deleting tablet b0542325c80241fca7e317cda64302a9
0507 08:12:17.776304 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.783391 (+ 7087us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.950220 (+166829us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.098427 (+148207us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3416753}
I20240507 08:12:18.100930 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac28170d9c3648b79a4394c7e51334bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.101565 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e766383f2adc47e1affa46ec627d652a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.101740 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba5d52927ec748c6bc543ccb9c95cce6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.101972 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11feca1cf7924e08958a538b96883c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.102118 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1142) took 3717 ms. Trace:
I20240507 08:12:18.102118 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf0c092cf64d2786382d5e9c2b2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.102385 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85b9b4b5d2884fc993cfae9b41824146. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.102268 6513 rpcz_store.cc:276] 0507 08:12:14.384560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.384622 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:18.101324 (+3716702us) ts_tablet_manager.cc:1120] Deleting tablet ab350a3f4f3341c2ad0dda3ce054e199
0507 08:12:18.101332 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.102099 (+ 767us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3716514}
I20240507 08:12:18.102536 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0eed609cb24474592c5b66b63e7669f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.102759 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1143) took 3658 ms. Trace:
I20240507 08:12:18.102885 6513 rpcz_store.cc:276] 0507 08:12:14.444367 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.444429 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:18.102664 (+3658235us) ts_tablet_manager.cc:1120] Deleting tablet 55bacb9fd24b46709704eb852e051eb1
0507 08:12:18.102672 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.102743 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3655541}
I20240507 08:12:18.103169 6513 tablet_replica.cc:330] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:18.103375 6513 raft_consensus.cc:2229] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.103400 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b78ec5ea7c43149c89be8dd2df0e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.109514 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b0542325c80241fca7e317cda64302a9 (table default.Table_86003022cb914d9ead81ce912a28d1cd [id=657d76ac3c524ee6988bda523bbd19c3]) successfully deleted
W20240507 08:12:18.109905 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ab350a3f4f3341c2ad0dda3ce054e199 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab350a3f4f3341c2ad0dda3ce054e199 already in progress: creating tablet
W20240507 08:12:18.110240 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 55bacb9fd24b46709704eb852e051eb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 55bacb9fd24b46709704eb852e051eb1 already in progress: creating tablet
I20240507 08:12:18.111519 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decb2294f6cd4800b332eccab802d224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.112222 6513 raft_consensus.cc:2258] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.112792 6513 ts_tablet_manager.cc:1875] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.113670 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1165) took 2613 ms. Trace:
I20240507 08:12:18.113811 5841 rpcz_store.cc:276] 0507 08:12:15.500212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.500739 (+ 527us) service_pool.cc:225] Handling call
0507 08:12:17.799524 (+2298785us) ts_tablet_manager.cc:1120] Deleting tablet c707458d1c094d87b2204683b28b49e4
0507 08:12:17.799533 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:17.910959 (+111426us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:17.913977 (+ 3018us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.113651 (+199674us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2298378}
I20240507 08:12:18.114270 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c707458d1c094d87b2204683b28b49e4 (table default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]) successfully deleted
I20240507 08:12:18.114619 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1166) took 2445 ms. Trace:
I20240507 08:12:18.114732 5841 rpcz_store.cc:276] 0507 08:12:15.668808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.668866 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:18.114514 (+2445648us) ts_tablet_manager.cc:1120] Deleting tablet 62895377a8ac4506a2f4e10039dbdd74
0507 08:12:18.114523 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.114604 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2431765}
W20240507 08:12:18.115159 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 62895377a8ac4506a2f4e10039dbdd74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62895377a8ac4506a2f4e10039dbdd74 already in progress: creating tablet
I20240507 08:12:18.115447 5841 tablet_replica.cc:330] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.115655 5841 raft_consensus.cc:2229] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.115854 5841 raft_consensus.cc:2258] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.116851 5841 ts_tablet_manager.cc:1875] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.117175 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6d5dc177474ca32f9a3154a59e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.120096 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192492075dbf4badb8c70a9546ddd80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.124428 5843 tablet_bootstrap.cc:492] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:18.124539 5843 ts_tablet_manager.cc:1367] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.006s
I20240507 08:12:18.125144 5843 raft_consensus.cc:348] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.125325 5843 raft_consensus.cc:374] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.125358 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f859146c376c4b1eb52bff953f05837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.125373 5843 raft_consensus.cc:729] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.125623 5843 consensus_queue.cc:260] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.126226 5843 ts_tablet_manager.cc:1398] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:18.126600 5843 tablet_bootstrap.cc:492] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:18.127811 5843 tablet_bootstrap.cc:654] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.130360 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet f859146c376c4b1eb52bff953f05837b
I20240507 08:12:18.131593 5843 tablet_bootstrap.cc:492] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:18.132083 5843 ts_tablet_manager.cc:1367] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:12:18.133095 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf0c092cf64d2786382d5e9c2b2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.134688 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet f859146c376c4b1eb52bff953f05837b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.133004 5843 raft_consensus.cc:348] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.135421 5843 raft_consensus.cc:374] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.135610 5843 raft_consensus.cc:729] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.136057 5843 consensus_queue.cc:260] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.137061 5843 ts_tablet_manager.cc:1398] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:18.138020 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5f53576c761847f081c511b4d5c067ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:18.138298 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0eed609cb24474592c5b66b63e7669f. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.138806 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f53576c761847f081c511b4d5c067ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.139379 5681 tablet_service.cc:1514] Processing DeleteTablet for tablet f859146c376c4b1eb52bff953f05837b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.141449 7552 consensus_queue.cc:1035] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.142724 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c830acb24c5c899eecff2c6b8bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.144639 5843 tablet_bootstrap.cc:492] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:18.147225 5843 tablet_bootstrap.cc:654] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.147908 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 11feca1cf7924e08958a538b96883c21 (DEFAULT_TABLE table=default.Table_a00f64d4fd4146a4bad06d8aa738d872 [id=ae0430787a25415da564030210943d89]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.154343 7590 raft_consensus.cc:482] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.154482 7590 raft_consensus.cc:504] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.154735 5546 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d1a4e05273b4f8c83dc8c7825a39310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.155136 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 1baf0c092cf64d2786382d5e9c2b2e72 (DEFAULT_TABLE table=default.Table_3d79cddfa377454d99a02c32c919b6a5 [id=e5d732adf0834bd69e3e9b1d0c9fe408]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.155881 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e298f11ff9f54e48b8de435f6849f39a (table default.Table_4f4c310282004fbe930f8b2aa57ed2c0 [id=4fbc482494d84698b894110d97cb9edf]) successfully deleted
W20240507 08:12:18.156090 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ddcb720c09c5486995391837fc54e7ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb720c09c5486995391837fc54e7ce already in progress: creating tablet
W20240507 08:12:18.156272 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f9b60e7fd01c4c96911fff6bf62ade96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b60e7fd01c4c96911fff6bf62ade96 already in progress: creating tablet
I20240507 08:12:18.156708 5565 tablet_service.cc:1467] Processing CreateTablet for tablet 0558c830acb24c5c899eecff2c6b8bff (DEFAULT_TABLE table=default.Table_edb3317ccbd440c99457c0d12ce3c25c [id=1f500681aaac43a7b13b2c4a369e7bd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.157503 5549 tablet_service.cc:1467] Processing CreateTablet for tablet d0eed609cb24474592c5b66b63e7669f (DEFAULT_TABLE table=default.Table_454ae1a1af424a77a79ea0010b86a0bb [id=d191061a79464d52afb0198638bdf7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.157920 5840 tablet_bootstrap.cc:492] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:18.158001 5654 consensus_peers.cc:485] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.166363 7590 leader_election.cc:288] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.053058 7602 raft_consensus.cc:482] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.166963 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 636e8f123b9540f3afedf02e4df5dfb5
I20240507 08:12:18.168020 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:18.170606 5520 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 28e5a380f2784e3a806afcbb5cd2519a
I20240507 08:12:18.171949 5553 tablet_service.cc:1467] Processing CreateTablet for tablet c4b78ec5ea7c43149c89be8dd2df0e45 (DEFAULT_TABLE table=default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6 [id=6d327ba6d58a4d2caed8b602db929fe8]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.172380 5546 tablet_service.cc:1467] Processing CreateTablet for tablet 192492075dbf4badb8c70a9546ddd80a (DEFAULT_TABLE table=default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.174870 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1145) took 2196 ms. Trace:
I20240507 08:12:18.174969 7079 rpcz_store.cc:276] 0507 08:12:15.978840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.979145 (+ 305us) service_pool.cc:225] Handling call
0507 08:12:18.082609 (+2103464us) ts_tablet_manager.cc:1120] Deleting tablet 38fc0180e9d34d11885b819529407930
0507 08:12:18.082616 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.174851 (+ 92235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103307}
I20240507 08:12:18.176339 5557 tablet_service.cc:1514] Processing DeleteTablet for tablet f859146c376c4b1eb52bff953f05837b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:18.176795 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 38fc0180e9d34d11885b819529407930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38fc0180e9d34d11885b819529407930 already in progress: creating tablet
W20240507 08:12:18.177680 5378 consensus_peers.cc:485] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.178092 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f13cbf54f8414b51bc0105896b80f077"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.178735 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f13cbf54f8414b51bc0105896b80f077 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:18.180399 5378 consensus_peers.cc:485] T accf88112b0b408e8d24828fc93055ef P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.181439 5841 ts_tablet_manager.cc:1888] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:18.185308 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d2ac4a0f8696476aab2548d83435ddc6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.185823 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2ac4a0f8696476aab2548d83435ddc6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:18.187048 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0558c830acb24c5c899eecff2c6b8bff consensus state has no leader
I20240507 08:12:18.187356 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bf3bbe27b82248878e70d707911633bc" } new_table_name: "default.Table_ec6c5b0ac0274d1fb210bcf62f46df8d" modify_external_catalogs: true
I20240507 08:12:18.187543 7079 tablet_replica.cc:330] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:18.187719 7079 raft_consensus.cc:2229] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.187845 7079 raft_consensus.cc:2258] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.188689 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9383fc310d0f4a6fb20a0c555d0e8257" } modify_external_catalogs: true
I20240507 08:12:18.188840 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9383fc310d0f4a6fb20a0c555d0e8257" } modify_external_catalogs: true
W20240507 08:12:18.189342 5517 leader_election.cc:341] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 28e5a380f2784e3a806afcbb5cd2519a
I20240507 08:12:18.194231 5554 tablet_service.cc:1467] Processing CreateTablet for tablet 461a6d5dc177474ca32f9a3154a59e92 (DEFAULT_TABLE table=default.Table_8d8d78277b8044a7b75cc3409a82ef21 [id=f7e6ac0a045d4b2f91debde6fa3be7ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.194576 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.194736 5517 leader_election.cc:302] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:18.198418 5562 tablet_service.cc:1467] Processing CreateTablet for tablet f859146c376c4b1eb52bff953f05837b (DEFAULT_TABLE table=default.Table_6cc465fb947d406f85b3779471ec43cc [id=81c7c926583347188ad733e695b06d33]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.198899 5560 tablet_service.cc:1467] Processing CreateTablet for tablet decb2294f6cd4800b332eccab802d224 (DEFAULT_TABLE table=default.Table_7ab1a575a68349d4a23fb56a2fd5098b [id=ab2c9bf1437b454f8d30a179d61fdea7]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:18.199347 5654 leader_election.cc:341] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.200139 7590 raft_consensus.cc:2726] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.200462 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet 47a154769d0f4501837888feba98f33d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.201154 7079 ts_tablet_manager.cc:1875] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.201331 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f35aa593fc4c45319217c4ce761d30f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:18.201782 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1baf0c092cf64d2786382d5e9c2b2e72. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.201962 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c4b78ec5ea7c43149c89be8dd2df0e45. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.201934 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f35aa593fc4c45319217c4ce761d30f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.202147 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0eed609cb24474592c5b66b63e7669f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.202288 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e766383f2adc47e1affa46ec627d652a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.202791 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0558c830acb24c5c899eecff2c6b8bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.202946 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet decb2294f6cd4800b332eccab802d224. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.203143 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f859146c376c4b1eb52bff953f05837b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.203750 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 461a6d5dc177474ca32f9a3154a59e92. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.204154 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 192492075dbf4badb8c70a9546ddd80a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.199234 7602 raft_consensus.cc:504] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:18.206837 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 83ed63048286415cadae77b750923f0d consensus state has no leader
I20240507 08:12:18.208837 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11feca1cf7924e08958a538b96883c21. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.207353 7602 leader_election.cc:288] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.211092 5699 tablet_service.cc:1514] Processing DeleteTablet for tablet 636e8f123b9540f3afedf02e4df5dfb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:17 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:18.220175 5517 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:18.234306 6675 raft_consensus.cc:482] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.234473 6675 raft_consensus.cc:504] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
W20240507 08:12:18.243983 5520 leader_election.cc:341] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:18.244082 6675 leader_election.cc:288] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:18.244103 5520 leader_election.cc:302] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:18.254524 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ff84c856a98f49c1987c5064a57e91bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.255177 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff84c856a98f49c1987c5064a57e91bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.258456 5840 tablet_bootstrap.cc:654] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.259905 6675 consensus_queue.cc:1035] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.260835 7602 raft_consensus.cc:2726] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.236307 7552 consensus_queue.cc:1035] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.239284 7550 raft_consensus.cc:482] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.272190 7550 raft_consensus.cc:504] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.273005 7550 leader_election.cc:288] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.247648 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.273391 5446 raft_consensus.cc:2454] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:18.274385 5841 log.cc:1196] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/975f22194aff40609a3cfa63b9b3aca2
I20240507 08:12:18.274395 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.274704 5656 leader_election.cc:302] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.275061 6675 raft_consensus.cc:2781] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:18.275144 6675 raft_consensus.cc:482] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20240507 08:12:18.275141 5378 leader_election.cc:341] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.275209 6675 raft_consensus.cc:3037] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.278930 6675 raft_consensus.cc:504] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.279975 6675 leader_election.cc:288] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:12:18.280287 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5370d022aa904900a28842b887b13187 consensus state has no leader
I20240507 08:12:18.280962 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:18.281183 5446 raft_consensus.cc:3037] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.283978 5446 raft_consensus.cc:2454] T fed715ba993a4f0c9e54b209ef031981 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:18.284380 5656 leader_election.cc:302] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.286965 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_77bee6f5db85437bb6ecccd959ba929f" } modify_external_catalogs: true
I20240507 08:12:18.287194 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_77bee6f5db85437bb6ecccd959ba929f" } modify_external_catalogs: true
I20240507 08:12:18.287536 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.287803 7550 raft_consensus.cc:482] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.287952 7550 raft_consensus.cc:504] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.288702 7550 leader_election.cc:288] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.289563 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.289635 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fed715ba993a4f0c9e54b209ef031981" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:18.290539 5378 leader_election.cc:341] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.290589 7358 raft_consensus.cc:2781] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:18.290699 7358 raft_consensus.cc:686] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:18.290930 7358 consensus_queue.cc:237] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.275156 5841 ts_tablet_manager.cc:1909] T 975f22194aff40609a3cfa63b9b3aca2 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.295526 7079 ts_tablet_manager.cc:1888] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:18.295661 7079 log.cc:1196] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b2df872d04b94c30872b03367df3cc31
I20240507 08:12:18.296017 7079 ts_tablet_manager.cc:1909] T b2df872d04b94c30872b03367df3cc31 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:18.296332 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1172) took 2064 ms. Trace:
I20240507 08:12:18.296434 7079 rpcz_store.cc:276] 0507 08:12:16.232246 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.237746 (+ 5500us) service_pool.cc:225] Handling call
0507 08:12:18.175194 (+1937448us) ts_tablet_manager.cc:1120] Deleting tablet b2df872d04b94c30872b03367df3cc31
0507 08:12:18.187494 (+ 12300us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.255599 (+ 68105us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.295406 (+ 39807us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.296314 (+ 908us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1937293}
I20240507 08:12:18.296939 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b2df872d04b94c30872b03367df3cc31 (table default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]) successfully deleted
I20240507 08:12:18.297530 7550 raft_consensus.cc:482] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.297685 7550 raft_consensus.cc:504] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.298278 7550 leader_election.cc:288] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.301333 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0f1645256ab34e18a40eb96a15cac1de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.302069 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f1645256ab34e18a40eb96a15cac1de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.306052 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:18.306576 5378 leader_election.cc:341] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.309729 5715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:18.310081 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:18.309760 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:18.310109 5654 leader_election.cc:341] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.314147 5840 tablet_bootstrap.cc:492] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:18.314379 5840 ts_tablet_manager.cc:1367] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.157s user 0.007s sys 0.002s
I20240507 08:12:18.315351 5840 raft_consensus.cc:348] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.315822 5840 raft_consensus.cc:374] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.315937 5840 raft_consensus.cc:729] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.316273 5840 consensus_queue.cc:260] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.317080 5840 ts_tablet_manager.cc:1398] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:18.317497 5840 tablet_bootstrap.cc:492] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:18.318914 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1167) took 2646 ms. Trace:
I20240507 08:12:18.319063 5841 rpcz_store.cc:276] 0507 08:12:15.672882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.672953 (+ 71us) service_pool.cc:225] Handling call
0507 08:12:18.115409 (+2442456us) ts_tablet_manager.cc:1120] Deleting tablet 975f22194aff40609a3cfa63b9b3aca2
0507 08:12:18.115417 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.154213 (+ 38796us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.181308 (+ 27095us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.318894 (+137586us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2432542}
I20240507 08:12:18.319238 5840 tablet_bootstrap.cc:654] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.319720 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]) successfully deleted
I20240507 08:12:18.320112 5841 tablet_replica.cc:330] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.320358 5841 raft_consensus.cc:2229] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:18.314155 5654 leader_election.cc:341] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.320611 5841 raft_consensus.cc:2258] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:18.320986 5377 leader_election.cc:341] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.321085 5377 leader_election.cc:302] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:18.321594 5377 leader_election.cc:341] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.321700 5377 leader_election.cc:302] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:18.322086 5377 leader_election.cc:341] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.322176 5377 leader_election.cc:302] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.322598 7552 raft_consensus.cc:2726] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.322810 7552 raft_consensus.cc:2726] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.322903 7552 raft_consensus.cc:2726] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.327504 5841 ts_tablet_manager.cc:1875] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.330384 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1168) took 2326 ms. Trace:
I20240507 08:12:18.330482 7079 rpcz_store.cc:276] 0507 08:12:16.003452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.227763 (+224311us) service_pool.cc:225] Handling call
0507 08:12:18.330254 (+2102491us) ts_tablet_manager.cc:1120] Deleting tablet 0ddb7dd28f43488581d3a75bf1d68a8a
0507 08:12:18.330265 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.330365 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2069256}
W20240507 08:12:18.330890 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 0ddb7dd28f43488581d3a75bf1d68a8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddb7dd28f43488581d3a75bf1d68a8a already in progress: creating tablet
I20240507 08:12:18.330821 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a3ed2876b6214485a0f2bbfb0d196d5a" } new_table_name: "default.Table_358ff4deeaf343de9c528cf8166e72b0" modify_external_catalogs: true
I20240507 08:12:18.332777 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1169) took 2329 ms. Trace:
I20240507 08:12:18.333676 7079 rpcz_store.cc:276] 0507 08:12:16.003510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.230014 (+226504us) service_pool.cc:225] Handling call
0507 08:12:18.332639 (+2102625us) ts_tablet_manager.cc:1120] Deleting tablet b15c8ad2a8324c75a1ef49e219b962cf
0507 08:12:18.332652 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.332747 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2071546}
I20240507 08:12:18.338209 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1170) took 2334 ms. Trace:
I20240507 08:12:18.338971 7079 rpcz_store.cc:276] 0507 08:12:16.003577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.230746 (+227169us) service_pool.cc:225] Handling call
0507 08:12:18.338093 (+2107347us) ts_tablet_manager.cc:1120] Deleting tablet c20c60fe0aad4cb3b12d04ad1a87d9f4
0507 08:12:18.338104 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.338193 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2076933}
W20240507 08:12:18.339404 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b15c8ad2a8324c75a1ef49e219b962cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b15c8ad2a8324c75a1ef49e219b962cf already in progress: creating tablet
W20240507 08:12:18.340715 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 already in progress: creating tablet
I20240507 08:12:18.346153 7079 tablet_replica.cc:330] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:18.346386 7079 raft_consensus.cc:2229] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.346658 7079 raft_consensus.cc:2258] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.347544 7079 ts_tablet_manager.cc:1875] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.354820 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2f247862fcb486e9776cf2e1b2107bb" } modify_external_catalogs: true
I20240507 08:12:18.355095 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2f247862fcb486e9776cf2e1b2107bb" } modify_external_catalogs: true
I20240507 08:12:18.360021 5091 master_runner.cc:186] Error getting master registration for 127.4.246.253:43459: OK, instance_id { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" instance_seqno: 1715069530908792 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20240507 08:12:18.360265 5091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20240507 08:12:18.368250 5654 consensus_peers.cc:485] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.373492 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
I20240507 08:12:18.375437 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbe582048e334931902cc697661690ca" } modify_external_catalogs: true
I20240507 08:12:18.375988 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbe582048e334931902cc697661690ca" } modify_external_catalogs: true
W20240507 08:12:18.378140 5656 consensus_peers.cc:485] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.381503 7100 raft_consensus.cc:482] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.381799 7100 raft_consensus.cc:504] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:18.382333 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7202cad072874dfcad0a9ad563a0bde4 has no consensus state
I20240507 08:12:18.382923 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.383324 5445 raft_consensus.cc:2454] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:18.383862 7499 raft_consensus.cc:482] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.383973 7499 raft_consensus.cc:504] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.384346 5839 tablet_bootstrap.cc:492] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:18.384511 5839 ts_tablet_manager.cc:1367] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.406s user 0.006s sys 0.009s
I20240507 08:12:18.384523 7499 leader_election.cc:288] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.385715 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:18.386159 5378 leader_election.cc:341] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.386583 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.387459 5656 leader_election.cc:302] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:18.387615 5654 leader_election.cc:341] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.387812 6675 raft_consensus.cc:2781] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240507 08:12:18.387889 5654 consensus_peers.cc:485] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.387933 6675 raft_consensus.cc:482] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:18.388113 6675 raft_consensus.cc:3037] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.388736 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e5179c5e5c224b3690b690b725a01cd0" } modify_external_catalogs: true
I20240507 08:12:18.388972 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e5179c5e5c224b3690b690b725a01cd0" } modify_external_catalogs: true
I20240507 08:12:18.390590 6675 raft_consensus.cc:504] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.395138 6675 leader_election.cc:288] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.395344 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:18.395586 5443 raft_consensus.cc:3037] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.395571 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:18.395845 7100 leader_election.cc:288] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.397099 7394 consensus_queue.cc:1035] T aca8c38841db45e38ae5d8d965a16d0e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.398742 5443 raft_consensus.cc:2454] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:18.399152 5656 leader_election.cc:302] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.399502 7358 raft_consensus.cc:2781] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:18.399639 7358 raft_consensus.cc:686] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:18.399932 7358 consensus_queue.cc:237] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.404347 5841 ts_tablet_manager.cc:1888] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:18.404549 5841 log.cc:1196] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/1ab94b1a09dd4969b928b017229ce4bf
I20240507 08:12:18.404915 5841 ts_tablet_manager.cc:1909] T 1ab94b1a09dd4969b928b017229ce4bf P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.405332 5840 tablet_bootstrap.cc:492] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:18.407228 5840 ts_tablet_manager.cc:1367] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.002s
I20240507 08:12:18.409016 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1168) took 2731 ms. Trace:
I20240507 08:12:18.409289 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b60ec15a83934999871425ef9a64ec72" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:18.410524 5377 leader_election.cc:341] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.410650 5377 leader_election.cc:302] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.411139 7499 raft_consensus.cc:2726] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.411123 5582 raft_consensus.cc:3037] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.409492 5841 rpcz_store.cc:276] 0507 08:12:15.677534 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.677619 (+ 85us) service_pool.cc:225] Handling call
0507 08:12:18.320052 (+2642433us) ts_tablet_manager.cc:1120] Deleting tablet 1ab94b1a09dd4969b928b017229ce4bf
0507 08:12:18.320065 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.331077 (+ 11012us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.404236 (+ 73159us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.408999 (+ 4763us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2624438}
I20240507 08:12:18.411458 5841 tablet_replica.cc:330] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.411654 5841 raft_consensus.cc:2229] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:18.413728 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 1ab94b1a09dd4969b928b017229ce4bf (table default.Table_3ca80faa43124b57991879a94bd046cd [id=d26d851d0f5244419cc308132e26e5e1]) successfully deleted
I20240507 08:12:18.425675 5840 raft_consensus.cc:348] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.426079 5840 raft_consensus.cc:374] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.426375 5840 raft_consensus.cc:729] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.426904 5840 consensus_queue.cc:260] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.426952 5839 raft_consensus.cc:348] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.427151 5839 raft_consensus.cc:374] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.427947 5839 raft_consensus.cc:729] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.429381 5839 consensus_queue.cc:260] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.430094 5841 raft_consensus.cc:2258] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.430259 6513 ts_tablet_manager.cc:1888] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:18.430455 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30b57b7f7f384fda8a21c5bbd95dcdd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:18.430961 5841 ts_tablet_manager.cc:1875] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.431177 6513 log.cc:1196] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4306751d16444ec5948c2ba5831063f4
I20240507 08:12:18.431182 5839 ts_tablet_manager.cc:1398] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
W20240507 08:12:18.431205 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30b57b7f7f384fda8a21c5bbd95dcdd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.431463 5839 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:18.431577 6513 ts_tablet_manager.cc:1909] T 4306751d16444ec5948c2ba5831063f4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:18.432968 5839 tablet_bootstrap.cc:654] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.437129 7550 raft_consensus.cc:482] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.437248 7550 raft_consensus.cc:504] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.437808 7550 leader_election.cc:288] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.444967 5255 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 449d6c8270184ce2918cee8ce731f788 (table default.Table_2c4d993ae928415181f53571b6bf0815 [id=3764a23f5a7847c8b46385bdfaf51f20]). Expected version 1 got 0
I20240507 08:12:18.448735 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 454c4c492a204c67bc6f48b7ae6cf054 (DEFAULT_TABLE table=default.Table_e8d09d71d5ff4cd68efa2496e75c47d7 [id=428286324c514b788c20ebf09278670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.449132 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454c4c492a204c67bc6f48b7ae6cf054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.450273 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 047497b14ec14209b5399b1755424a78 (DEFAULT_TABLE table=default.Table_f1cc86864eaa4b64a5b7de5c559b25a9 [id=d1a6c48f7e744081a4dee011012920f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.450568 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047497b14ec14209b5399b1755424a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.452374 5412 tablet_service.cc:1467] Processing CreateTablet for tablet a2adde63d55e46158a80be0c16667963 (DEFAULT_TABLE table=default.Table_7c4a61a066d64809ada26672081b8ed7 [id=08c0fc5abf4b4c559de276d16c041566]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.452678 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2adde63d55e46158a80be0c16667963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.453801 5424 tablet_service.cc:1467] Processing CreateTablet for tablet dbf6bca9f2df4882962cab98fa1a11cf (DEFAULT_TABLE table=default.Table_14a9319948f348828122c47d5271ab38 [id=d9fc97eb407d49d495d10e8104179262]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.454128 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf6bca9f2df4882962cab98fa1a11cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.459596 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 417af04d2e5f48358b437ac2cf00bad6 (DEFAULT_TABLE table=default.Table_9bfb20b9d7454bc2b58d6c328b6ecc47 [id=dd8c12fd572c4326b305220564b552a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.460029 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417af04d2e5f48358b437ac2cf00bad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.460984 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1145) took 3738 ms. Trace:
I20240507 08:12:18.461081 6513 rpcz_store.cc:276] 0507 08:12:14.722901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.723107 (+ 206us) service_pool.cc:225] Handling call
0507 08:12:18.103136 (+3380029us) ts_tablet_manager.cc:1120] Deleting tablet 4306751d16444ec5948c2ba5831063f4
0507 08:12:18.103143 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.233283 (+130140us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.412232 (+178949us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.460954 (+ 48722us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23782,"tablet-delete.queue_time_us":3377779}
I20240507 08:12:18.461520 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1147) took 3608 ms. Trace:
I20240507 08:12:18.461606 6513 rpcz_store.cc:276] 0507 08:12:14.853168 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.853488 (+ 320us) service_pool.cc:225] Handling call
0507 08:12:18.461387 (+3607899us) ts_tablet_manager.cc:1120] Deleting tablet 093fb3986a674b93a435408c7ec85500
0507 08:12:18.461399 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.461504 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3606039}
I20240507 08:12:18.461906 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1148) took 3555 ms. Trace:
I20240507 08:12:18.461968 6513 rpcz_store.cc:276] 0507 08:12:14.906420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:14.906476 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:18.461814 (+3555338us) ts_tablet_manager.cc:1120] Deleting tablet 19b77b6ef778454e91105e1e9b60020d
0507 08:12:18.461824 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.461894 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3555129}
I20240507 08:12:18.462217 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1149) took 3423 ms. Trace:
I20240507 08:12:18.462273 6513 rpcz_store.cc:276] 0507 08:12:15.038319 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.038370 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:18.462141 (+3423771us) ts_tablet_manager.cc:1120] Deleting tablet 62d6eac6baa249eb847c5625d64f998f
0507 08:12:18.462148 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.462207 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3423622}
I20240507 08:12:18.462518 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1154) took 3380 ms. Trace:
I20240507 08:12:18.462574 6513 rpcz_store.cc:276] 0507 08:12:15.082222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.082590 (+ 368us) service_pool.cc:225] Handling call
0507 08:12:18.462437 (+3379847us) ts_tablet_manager.cc:1120] Deleting tablet a24813c72abd41c3830c7392c6149151
0507 08:12:18.462445 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.462506 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3379695}
I20240507 08:12:18.462865 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1155) took 3380 ms. Trace:
I20240507 08:12:18.464974 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 738f7dc56eb34439ad42361ff12290be (DEFAULT_TABLE table=default.Table_028d1e1e16014b8986c0bfebf632a933 [id=b01e59c014414fc4b27da0d66a2dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.465302 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738f7dc56eb34439ad42361ff12290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.467454 5418 tablet_service.cc:1467] Processing CreateTablet for tablet b069d601b8fb4e75934675e7996a683a (DEFAULT_TABLE table=default.Table_2311302968aa441c8cfc10afe6e08044 [id=5222dfd0213148b1aa95cec0abb95b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.467769 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b069d601b8fb4e75934675e7996a683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.468232 5410 tablet_service.cc:1467] Processing CreateTablet for tablet cbe127ab59f04e18a6b1b5bd30dd5d38 (DEFAULT_TABLE table=default.Table_5a685699b57943ee8abbfa913c98aa95 [id=d7741ac0257c491c9395150eb5fa345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.468183 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 587b0026e23143dda4e38ba9460fbf62 (DEFAULT_TABLE table=default.Table_7cf76b6dcab2476387df464cd9bb0bcb [id=75a3f7817e03415b8408f10d4a8f54ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.469403 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587b0026e23143dda4e38ba9460fbf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.469470 6513 rpcz_store.cc:276] 0507 08:12:15.082317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.082772 (+ 455us) service_pool.cc:225] Handling call
0507 08:12:18.462770 (+3379998us) ts_tablet_manager.cc:1120] Deleting tablet db92da02e5af49f5a13a814f3ce4f9c3
0507 08:12:18.462780 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.462851 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3379880}
I20240507 08:12:18.469834 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1167) took 3259 ms. Trace:
I20240507 08:12:18.469904 6513 rpcz_store.cc:276] 0507 08:12:15.210377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.227891 (+ 17514us) service_pool.cc:225] Handling call
0507 08:12:18.469715 (+3241824us) ts_tablet_manager.cc:1120] Deleting tablet cc798eb4e02c456880ef3014ebbb7e24
0507 08:12:18.469727 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.469819 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3241577}
I20240507 08:12:18.470170 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1168) took 3239 ms. Trace:
I20240507 08:12:18.470228 6513 rpcz_store.cc:276] 0507 08:12:15.231745 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.231793 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:18.470086 (+3238293us) ts_tablet_manager.cc:1120] Deleting tablet f748727d3a88442d90cf81bae48acd36
0507 08:12:18.470093 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.470158 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3238112}
I20240507 08:12:18.470466 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1181) took 2955 ms. Trace:
I20240507 08:12:18.470527 6513 rpcz_store.cc:276] 0507 08:12:15.514740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.522124 (+ 7384us) service_pool.cc:225] Handling call
0507 08:12:18.470401 (+2948277us) ts_tablet_manager.cc:1120] Deleting tablet 0d617213b959452ab6519809e56db128
0507 08:12:18.470408 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.470454 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2948087}
I20240507 08:12:18.470733 6513 tablet_replica.cc:330] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:18.470932 6513 raft_consensus.cc:2229] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:18.472548 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:18.472556 5654 leader_election.cc:341] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.475973 5840 ts_tablet_manager.cc:1398] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.068s user 0.001s sys 0.001s
I20240507 08:12:18.476243 5840 tablet_bootstrap.cc:492] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:18.477710 5840 tablet_bootstrap.cc:654] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.477860 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe127ab59f04e18a6b1b5bd30dd5d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.479033 7499 raft_consensus.cc:482] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.479152 7499 raft_consensus.cc:504] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.479589 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4306751d16444ec5948c2ba5831063f4 (table default.Table_1c35bc706b22451f90611a3e7bc81f4d [id=a005b1852f254fd183b3323e3d5ac6a3]) successfully deleted
I20240507 08:12:18.479946 5843 tablet_bootstrap.cc:492] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:18.479979 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 093fb3986a674b93a435408c7ec85500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 093fb3986a674b93a435408c7ec85500 already in progress: creating tablet
I20240507 08:12:18.479993 7499 leader_election.cc:288] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:18.480893 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 19b77b6ef778454e91105e1e9b60020d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19b77b6ef778454e91105e1e9b60020d already in progress: creating tablet
W20240507 08:12:18.481554 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 62d6eac6baa249eb847c5625d64f998f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62d6eac6baa249eb847c5625d64f998f already in progress: creating tablet
W20240507 08:12:18.482254 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a24813c72abd41c3830c7392c6149151 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a24813c72abd41c3830c7392c6149151 already in progress: creating tablet
I20240507 08:12:18.482477 5417 tablet_service.cc:1467] Processing CreateTablet for tablet aa3c303201df44bc84ec9de4fde58302 (DEFAULT_TABLE table=default.Table_0dba2acabcc144e5929e28d0332ed1df [id=cf6e228624bf4e049cfc5ce50c37a8e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:18.482635 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet db92da02e5af49f5a13a814f3ce4f9c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet db92da02e5af49f5a13a814f3ce4f9c3 already in progress: creating tablet
W20240507 08:12:18.482976 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet cc798eb4e02c456880ef3014ebbb7e24 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc798eb4e02c456880ef3014ebbb7e24 already in progress: creating tablet
I20240507 08:12:18.483085 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3c303201df44bc84ec9de4fde58302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.483839 6513 raft_consensus.cc:2258] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.484596 6513 ts_tablet_manager.cc:1875] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.485627 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ca3fc45914629801bec17d26ae40a (DEFAULT_TABLE table=default.Table_38b43849501546438b4a99226360030b [id=3748ab7e38d94b7598f2011a6056b70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.486658 5843 ts_tablet_manager.cc:1367] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.342s user 0.103s sys 0.105s
I20240507 08:12:18.488009 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee7ab58a652468dbc1ea742eb61bf86 (DEFAULT_TABLE table=default.Table_d54c9b59e4684e0482f46ffb7801ee02 [id=40c66f0c234b4590bece534cfc5a6243]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.488354 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 81a46b5a4afb470fa3400e9a7b43b3b4 (DEFAULT_TABLE table=default.Table_a8cdec0cbd5e4a049ec36a0f9c01efe3 [id=3912ea4db94d4fbab528f6834ddfeee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.489866 5414 tablet_service.cc:1467] Processing CreateTablet for tablet aae2f13901a146b7a2019709b28930e9 (DEFAULT_TABLE table=default.Table_ec50c8f7242944919aadbff5d3b2a54a [id=c2f82cf4d45c45c998c9499c5bbd735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.438598 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.473850 5546 tablet_service.cc:1467] Processing CreateTablet for tablet dbf6bca9f2df4882962cab98fa1a11cf (DEFAULT_TABLE table=default.Table_14a9319948f348828122c47d5271ab38 [id=d9fc97eb407d49d495d10e8104179262]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.474074 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 454c4c492a204c67bc6f48b7ae6cf054 (DEFAULT_TABLE table=default.Table_e8d09d71d5ff4cd68efa2496e75c47d7 [id=428286324c514b788c20ebf09278670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.492930 5546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf6bca9f2df4882962cab98fa1a11cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.493095 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454c4c492a204c67bc6f48b7ae6cf054. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.493100 5378 leader_election.cc:341] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.475828 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 047497b14ec14209b5399b1755424a78 (DEFAULT_TABLE table=default.Table_f1cc86864eaa4b64a5b7de5c559b25a9 [id=d1a6c48f7e744081a4dee011012920f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.493829 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047497b14ec14209b5399b1755424a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.495759 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 07f3d8a01876406faec5ddf3387c0870 (DEFAULT_TABLE table=default.Table_99c89f1c9f494e4786dba165357d1b31 [id=316dfca7af484c2a85fd5d083f9ebab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.496042 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f3d8a01876406faec5ddf3387c0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.476112 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 417af04d2e5f48358b437ac2cf00bad6 (DEFAULT_TABLE table=default.Table_9bfb20b9d7454bc2b58d6c328b6ecc47 [id=dd8c12fd572c4326b305220564b552a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.497843 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417af04d2e5f48358b437ac2cf00bad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.497813 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 863c67d4ad97490eb6179ad65113bd4b (DEFAULT_TABLE table=default.Table_fca4a8f0d4e244c0872dd877f3e6aec8 [id=6d9d8819c44c41e0bfdb60f3f8ebd275]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.476382 5555 tablet_service.cc:1467] Processing CreateTablet for tablet 738f7dc56eb34439ad42361ff12290be (DEFAULT_TABLE table=default.Table_028d1e1e16014b8986c0bfebf632a933 [id=b01e59c014414fc4b27da0d66a2dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.476660 5565 tablet_service.cc:1467] Processing CreateTablet for tablet cbe127ab59f04e18a6b1b5bd30dd5d38 (DEFAULT_TABLE table=default.Table_5a685699b57943ee8abbfa913c98aa95 [id=d7741ac0257c491c9395150eb5fa345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.498559 5551 tablet_service.cc:1467] Processing CreateTablet for tablet 72753784f6ea4166affb413877903e58 (DEFAULT_TABLE table=default.Table_54786977c4974775aaa70909567f71fe [id=3d5211ad8e2349b9b7b2d76c195d68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.476938 5554 tablet_service.cc:1467] Processing CreateTablet for tablet b069d601b8fb4e75934675e7996a683a (DEFAULT_TABLE table=default.Table_2311302968aa441c8cfc10afe6e08044 [id=5222dfd0213148b1aa95cec0abb95b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.477232 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 587b0026e23143dda4e38ba9460fbf62 (DEFAULT_TABLE table=default.Table_7cf76b6dcab2476387df464cd9bb0bcb [id=75a3f7817e03415b8408f10d4a8f54ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.477519 5548 tablet_service.cc:1467] Processing CreateTablet for tablet aa3c303201df44bc84ec9de4fde58302 (DEFAULT_TABLE table=default.Table_0dba2acabcc144e5929e28d0332ed1df [id=cf6e228624bf4e049cfc5ce50c37a8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.477815 5550 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ca3fc45914629801bec17d26ae40a (DEFAULT_TABLE table=default.Table_38b43849501546438b4a99226360030b [id=3748ab7e38d94b7598f2011a6056b70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.478149 5561 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee7ab58a652468dbc1ea742eb61bf86 (DEFAULT_TABLE table=default.Table_d54c9b59e4684e0482f46ffb7801ee02 [id=40c66f0c234b4590bece534cfc5a6243]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.478416 5557 tablet_service.cc:1467] Processing CreateTablet for tablet aae2f13901a146b7a2019709b28930e9 (DEFAULT_TABLE table=default.Table_ec50c8f7242944919aadbff5d3b2a54a [id=c2f82cf4d45c45c998c9499c5bbd735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.478663 5564 tablet_service.cc:1467] Processing CreateTablet for tablet 81a46b5a4afb470fa3400e9a7b43b3b4 (DEFAULT_TABLE table=default.Table_a8cdec0cbd5e4a049ec36a0f9c01efe3 [id=3912ea4db94d4fbab528f6834ddfeee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.500201 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863c67d4ad97490eb6179ad65113bd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.501595 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 454c4c492a204c67bc6f48b7ae6cf054 (DEFAULT_TABLE table=default.Table_e8d09d71d5ff4cd68efa2496e75c47d7 [id=428286324c514b788c20ebf09278670b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.501935 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 454c4c492a204c67bc6f48b7ae6cf054. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.502319 5559 tablet_service.cc:1467] Processing CreateTablet for tablet dc13f0fb39074793b41e7b3b0bc595e4 (DEFAULT_TABLE table=default.Table_830c53deefa541df8e6682e8c3ef976f [id=f2ce4de6ab6647d4a9426aba3c43bdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.502595 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc13f0fb39074793b41e7b3b0bc595e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.505379 5377 leader_election.cc:341] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.505488 5377 leader_election.cc:302] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.506865 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3c303201df44bc84ec9de4fde58302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.509064 5547 tablet_service.cc:1467] Processing CreateTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 (DEFAULT_TABLE table=default.Table_9fcb6776bc164f07ba3ad8154d10bf36 [id=1dc69fa11458492cac07f67e31af8a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.509361 5547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401402afc04f4a9082c02ca5bf39d6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.509466 5691 tablet_service.cc:1467] Processing CreateTablet for tablet dbf6bca9f2df4882962cab98fa1a11cf (DEFAULT_TABLE table=default.Table_14a9319948f348828122c47d5271ab38 [id=d9fc97eb407d49d495d10e8104179262]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.509666 5561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee7ab58a652468dbc1ea742eb61bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.509857 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbf6bca9f2df4882962cab98fa1a11cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.510145 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 6419b9be07564394a3b88ee36a14add3 (DEFAULT_TABLE table=default.Table_8fc4b266d274493fb54ebb06278ac446 [id=bad0680052bb41b48271e4330ccfd840]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.510437 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419b9be07564394a3b88ee36a14add3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.513669 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca3fc45914629801bec17d26ae40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.517879 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587b0026e23143dda4e38ba9460fbf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.519858 5557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae2f13901a146b7a2019709b28930e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.521848 5564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a46b5a4afb470fa3400e9a7b43b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.521728 5258 catalog_manager.cc:5484] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:18.524624 5565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe127ab59f04e18a6b1b5bd30dd5d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.525399 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 3f519b1ee222487a94e1625c6bb7c181 (DEFAULT_TABLE table=default.Table_cc67c2a61fe048a0acc93d29961dcdce [id=319412836ddf40908b62a2eab165511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.525904 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f519b1ee222487a94e1625c6bb7c181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.528335 5843 raft_consensus.cc:348] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.528539 5843 raft_consensus.cc:374] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.528600 5843 raft_consensus.cc:729] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.529114 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca3fc45914629801bec17d26ae40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.529623 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee7ab58a652468dbc1ea742eb61bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.530004 7550 raft_consensus.cc:2726] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.530341 5683 tablet_service.cc:1467] Processing CreateTablet for tablet a2adde63d55e46158a80be0c16667963 (DEFAULT_TABLE table=default.Table_7c4a61a066d64809ada26672081b8ed7 [id=08c0fc5abf4b4c559de276d16c041566]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.530472 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 047497b14ec14209b5399b1755424a78 (DEFAULT_TABLE table=default.Table_f1cc86864eaa4b64a5b7de5c559b25a9 [id=d1a6c48f7e744081a4dee011012920f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.530622 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a46b5a4afb470fa3400e9a7b43b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.530714 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2adde63d55e46158a80be0c16667963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.530953 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae2f13901a146b7a2019709b28930e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.478906 5549 tablet_service.cc:1467] Processing CreateTablet for tablet a2adde63d55e46158a80be0c16667963 (DEFAULT_TABLE table=default.Table_7c4a61a066d64809ada26672081b8ed7 [id=08c0fc5abf4b4c559de276d16c041566]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.480230 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.531992 5558 tablet_service.cc:1467] Processing CreateTablet for tablet 2414f836bbcb4c8ebd1d3ac8320308d6 (DEFAULT_TABLE table=default.Table_118c82c5ecc644d998c3d38c16544c42 [id=e814c21c288e442a8157eca400936250]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.532928 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 047497b14ec14209b5399b1755424a78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.535322 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac3765fa3484f88ad6fd930d289fd7a (DEFAULT_TABLE table=default.Table_f3940a8999cb4633b1d82922022afe2f [id=58525b203eb34740a2cc94330eb01be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.536698 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 417af04d2e5f48358b437ac2cf00bad6 (DEFAULT_TABLE table=default.Table_9bfb20b9d7454bc2b58d6c328b6ecc47 [id=dd8c12fd572c4326b305220564b552a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.537087 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 417af04d2e5f48358b437ac2cf00bad6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.539495 5689 tablet_service.cc:1467] Processing CreateTablet for tablet b069d601b8fb4e75934675e7996a683a (DEFAULT_TABLE table=default.Table_2311302968aa441c8cfc10afe6e08044 [id=5222dfd0213148b1aa95cec0abb95b79]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.539777 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b069d601b8fb4e75934675e7996a683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.540426 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 738f7dc56eb34439ad42361ff12290be (DEFAULT_TABLE table=default.Table_028d1e1e16014b8986c0bfebf632a933 [id=b01e59c014414fc4b27da0d66a2dde27]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.540696 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738f7dc56eb34439ad42361ff12290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.528776 5843 consensus_queue.cc:260] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.545821 5692 tablet_service.cc:1467] Processing CreateTablet for tablet cbe127ab59f04e18a6b1b5bd30dd5d38 (DEFAULT_TABLE table=default.Table_5a685699b57943ee8abbfa913c98aa95 [id=d7741ac0257c491c9395150eb5fa345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.545945 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 07f3d8a01876406faec5ddf3387c0870 (DEFAULT_TABLE table=default.Table_99c89f1c9f494e4786dba165357d1b31 [id=316dfca7af484c2a85fd5d083f9ebab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.546123 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cbe127ab59f04e18a6b1b5bd30dd5d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.546288 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f3d8a01876406faec5ddf3387c0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.482820 7636 tablet.cc:1722] T 2d45d93ed0d34806b9d9451c101f10af P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:18.547107 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 587b0026e23143dda4e38ba9460fbf62 (DEFAULT_TABLE table=default.Table_7cf76b6dcab2476387df464cd9bb0bcb [id=75a3f7817e03415b8408f10d4a8f54ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.547537 5554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b069d601b8fb4e75934675e7996a683a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.547530 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587b0026e23143dda4e38ba9460fbf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.547828 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2414f836bbcb4c8ebd1d3ac8320308d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.553105 5551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72753784f6ea4166affb413877903e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.557768 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 738f7dc56eb34439ad42361ff12290be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.557967 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 03069758ca5945aa9985aa9571ce2bfc (DEFAULT_TABLE table=default.Table_4d6a8373656e4d0da8b3529540c558af [id=71ac5228a8404365be05eda36fb60fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.558290 5840 tablet_bootstrap.cc:492] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:18.558413 5840 ts_tablet_manager.cc:1367] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.082s user 0.000s sys 0.002s
I20240507 08:12:18.559003 5840 raft_consensus.cc:348] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.559168 5840 raft_consensus.cc:374] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.559214 5840 raft_consensus.cc:729] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.559477 7602 raft_consensus.cc:482] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.559404 5840 consensus_queue.cc:260] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.559597 7602 raft_consensus.cc:504] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.559876 5840 ts_tablet_manager.cc:1398] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240507 08:12:18.560106 5840 tablet_bootstrap.cc:492] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:18.560142 7602 leader_election.cc:288] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.561131 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 4e7ca3fc45914629801bec17d26ae40a (DEFAULT_TABLE table=default.Table_38b43849501546438b4a99226360030b [id=3748ab7e38d94b7598f2011a6056b70f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.561213 5698 tablet_service.cc:1467] Processing CreateTablet for tablet aae2f13901a146b7a2019709b28930e9 (DEFAULT_TABLE table=default.Table_ec50c8f7242944919aadbff5d3b2a54a [id=c2f82cf4d45c45c998c9499c5bbd735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.561461 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e7ca3fc45914629801bec17d26ae40a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.561627 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aae2f13901a146b7a2019709b28930e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.561790 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 6ee7ab58a652468dbc1ea742eb61bf86 (DEFAULT_TABLE table=default.Table_d54c9b59e4684e0482f46ffb7801ee02 [id=40c66f0c234b4590bece534cfc5a6243]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.562063 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ee7ab58a652468dbc1ea742eb61bf86. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.563450 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2adde63d55e46158a80be0c16667963. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.566998 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 81a46b5a4afb470fa3400e9a7b43b3b4 (DEFAULT_TABLE table=default.Table_a8cdec0cbd5e4a049ec36a0f9c01efe3 [id=3912ea4db94d4fbab528f6834ddfeee9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.567293 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81a46b5a4afb470fa3400e9a7b43b3b4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.571285 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0d617213b959452ab6519809e56db128 because the tablet was not found. No further retry: Not found: Tablet not found: 0d617213b959452ab6519809e56db128
W20240507 08:12:18.573822 5378 leader_election.cc:341] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.575037 5260 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 2d45d93ed0d34806b9d9451c101f10af (table default.Table_1d12ff9c6bde45f7b0c2db06baf67cbc [id=6f00b78e5fce4f0181e4539ba6820c24]). Expected version 1 got 0
W20240507 08:12:18.575102 5517 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:12:18.575888 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_27c86e8536c344a7bf06b2f6b13b3082" } new_table_name: "default.Table_d5dadf7deaf043a8822529d3077f1f69" modify_external_catalogs: true
I20240507 08:12:18.582351 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:18.579736 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 4d99f656e19349c1935294fd7968ecd1 (DEFAULT_TABLE table=default.Table_b5153ea0c2884e12a0fbd2cb2fb5678a [id=b21deda92f9544ecada5c6bddc42b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.519860 5553 tablet_service.cc:1467] Processing CreateTablet for tablet adfa454f8e2b4c2386c63c3cf765b5b1 (DEFAULT_TABLE table=default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46 [id=31cf7b29628549ab95a414a28a7e0e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.584007 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac3765fa3484f88ad6fd930d289fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.584221 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 07f3d8a01876406faec5ddf3387c0870 (DEFAULT_TABLE table=default.Table_99c89f1c9f494e4786dba165357d1b31 [id=316dfca7af484c2a85fd5d083f9ebab6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.584597 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07f3d8a01876406faec5ddf3387c0870. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.585960 6675 raft_consensus.cc:482] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.586087 6675 raft_consensus.cc:504] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.586618 6675 leader_election.cc:288] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:18.587713 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be1b36ca5b6041d780e851f137d3a4c9 consensus state has no leader
I20240507 08:12:18.588943 5255 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_e3bcf078654d42ed8331d033b29e482c [id=ca5e7cb9fb9e41c8a4fda41af0c5144e] alter complete (version 2)
I20240507 08:12:18.590752 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 04a14cf04d7a4ce3a2df939d0db515b4
I20240507 08:12:18.591116 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 863c67d4ad97490eb6179ad65113bd4b (DEFAULT_TABLE table=default.Table_fca4a8f0d4e244c0872dd877f3e6aec8 [id=6d9d8819c44c41e0bfdb60f3f8ebd275]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.591470 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863c67d4ad97490eb6179ad65113bd4b. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.594022 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 138419us
I20240507 08:12:18.600955 5692 tablet_service.cc:1467] Processing CreateTablet for tablet aa3c303201df44bc84ec9de4fde58302 (DEFAULT_TABLE table=default.Table_0dba2acabcc144e5929e28d0332ed1df [id=cf6e228624bf4e049cfc5ce50c37a8e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.603247 7079 ts_tablet_manager.cc:1888] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:18.603318 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa3c303201df44bc84ec9de4fde58302. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.603502 7079 log.cc:1196] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/6b4c69d38e484f5c819b7dd915bf4410
I20240507 08:12:18.603912 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 (DEFAULT_TABLE table=default.Table_9fcb6776bc164f07ba3ad8154d10bf36 [id=1dc69fa11458492cac07f67e31af8a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.603996 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 6419b9be07564394a3b88ee36a14add3 (DEFAULT_TABLE table=default.Table_8fc4b266d274493fb54ebb06278ac446 [id=bad0680052bb41b48271e4330ccfd840]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.603996 5687 tablet_service.cc:1467] Processing CreateTablet for tablet adfa454f8e2b4c2386c63c3cf765b5b1 (DEFAULT_TABLE table=default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46 [id=31cf7b29628549ab95a414a28a7e0e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.604187 7079 ts_tablet_manager.cc:1909] T 6b4c69d38e484f5c819b7dd915bf4410 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:18.604539 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa454f8e2b4c2386c63c3cf765b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.604732 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401402afc04f4a9082c02ca5bf39d6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.607486 6675 raft_consensus.cc:482] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.607604 6675 raft_consensus.cc:504] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.608141 6675 leader_election.cc:288] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.609678 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419b9be07564394a3b88ee36a14add3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.610509 5689 tablet_service.cc:1467] Processing CreateTablet for tablet dc13f0fb39074793b41e7b3b0bc595e4 (DEFAULT_TABLE table=default.Table_830c53deefa541df8e6682e8c3ef976f [id=f2ce4de6ab6647d4a9426aba3c43bdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.610878 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc13f0fb39074793b41e7b3b0bc595e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.616672 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 4d99f656e19349c1935294fd7968ecd1 (DEFAULT_TABLE table=default.Table_b5153ea0c2884e12a0fbd2cb2fb5678a [id=b21deda92f9544ecada5c6bddc42b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.617007 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d99f656e19349c1935294fd7968ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.617808 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 3f519b1ee222487a94e1625c6bb7c181 (DEFAULT_TABLE table=default.Table_cc67c2a61fe048a0acc93d29961dcdce [id=319412836ddf40908b62a2eab165511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.618103 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f519b1ee222487a94e1625c6bb7c181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.620635 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 72753784f6ea4166affb413877903e58 (DEFAULT_TABLE table=default.Table_54786977c4974775aaa70909567f71fe [id=3d5211ad8e2349b9b7b2d76c195d68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.620929 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72753784f6ea4166affb413877903e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.623755 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 2414f836bbcb4c8ebd1d3ac8320308d6 (DEFAULT_TABLE table=default.Table_118c82c5ecc644d998c3d38c16544c42 [id=e814c21c288e442a8157eca400936250]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.624102 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2414f836bbcb4c8ebd1d3ac8320308d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.630477 5840 tablet_bootstrap.cc:654] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.631088 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d99f656e19349c1935294fd7968ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.631353 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 7202cad072874dfcad0a9ad563a0bde4 (DEFAULT_TABLE table=default.Table_ec6c5b0ac0274d1fb210bcf62f46df8d [id=452fa515268b42cebabfcd5f7573b4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.631657 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7202cad072874dfcad0a9ad563a0bde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.631851 5687 tablet_service.cc:1467] Processing CreateTablet for tablet e5f962ee766947fd83c7e016c1a901d1 (DEFAULT_TABLE table=default.Table_19a5c269746d46d7b76ea68c04acd280 [id=0353eb24b6b04bba924a2b0f33f61ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.632139 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f962ee766947fd83c7e016c1a901d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.633833 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 7202cad072874dfcad0a9ad563a0bde4 (DEFAULT_TABLE table=default.Table_ec6c5b0ac0274d1fb210bcf62f46df8d [id=452fa515268b42cebabfcd5f7573b4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.634191 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7202cad072874dfcad0a9ad563a0bde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.636126 5559 tablet_service.cc:1467] Processing CreateTablet for tablet e5f962ee766947fd83c7e016c1a901d1 (DEFAULT_TABLE table=default.Table_19a5c269746d46d7b76ea68c04acd280 [id=0353eb24b6b04bba924a2b0f33f61ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.636492 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03069758ca5945aa9985aa9571ce2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.636760 5552 tablet_service.cc:1467] Processing CreateTablet for tablet c74576ac62e04fb7a732f54e7b82e0ef (DEFAULT_TABLE table=default.Table_010a0902019b4e9381337fc5cd6a3834 [id=a46a6abbe82d4f078ddc784624b0218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.637128 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74576ac62e04fb7a732f54e7b82e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.637727 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 962144051ca3423d9bb8527229dd1b94 (DEFAULT_TABLE table=default.Table_289cce254034416eb97d18b1a1e6868f [id=185f9e52ad81487ead9068bb3d3ab534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.637866 5695 tablet_service.cc:1467] Processing CreateTablet for tablet 61255088a80b4c448b1b048e5ada5bb9 (DEFAULT_TABLE table=default.Table_eb2b1328f64f4a5d8ebdc6d68d4282a9 [id=43d81fcebfe1457ea9190ec20bb94ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.638041 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962144051ca3423d9bb8527229dd1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.638284 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61255088a80b4c448b1b048e5ada5bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.638475 5681 tablet_service.cc:1467] Processing CreateTablet for tablet e54273ed92f640d98b853a7661ae3e26 (DEFAULT_TABLE table=default.Table_bac149077ee54ee3ba57739c3b704a42 [id=362c05703392453ebeef62e6937a9139]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.638752 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54273ed92f640d98b853a7661ae3e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.641947 5688 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.642113 5687 tablet_service.cc:1467] Processing CreateTablet for tablet c74576ac62e04fb7a732f54e7b82e0ef (DEFAULT_TABLE table=default.Table_010a0902019b4e9381337fc5cd6a3834 [id=a46a6abbe82d4f078ddc784624b0218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.642395 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74576ac62e04fb7a732f54e7b82e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.644681 5377 leader_election.cc:341] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.646560 7602 raft_consensus.cc:482] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.646687 7602 raft_consensus.cc:504] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.647351 7602 leader_election.cc:288] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:18.647629 5654 consensus_peers.cc:485] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.648344 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2de26190bc44916809856d84ad68345" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.648639 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac3765fa3484f88ad6fd930d289fd7a (DEFAULT_TABLE table=default.Table_f3940a8999cb4633b1d82922022afe2f [id=58525b203eb34740a2cc94330eb01be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.648938 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac3765fa3484f88ad6fd930d289fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.648319 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ed7d7d2e8e424c90048c87c0fb0ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.651782 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:18.651952 5549 tablet_service.cc:1467] Processing CreateTablet for tablet e54273ed92f640d98b853a7661ae3e26 (DEFAULT_TABLE table=default.Table_bac149077ee54ee3ba57739c3b704a42 [id=362c05703392453ebeef62e6937a9139]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.652297 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54273ed92f640d98b853a7661ae3e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.652290 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 03069758ca5945aa9985aa9571ce2bfc (DEFAULT_TABLE table=default.Table_4d6a8373656e4d0da8b3529540c558af [id=71ac5228a8404365be05eda36fb60fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.652565 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03069758ca5945aa9985aa9571ce2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.653963 6513 ts_tablet_manager.cc:1888] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.654106 6513 log.cc:1196] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b1fadbfd330b4e699e151f83125da281
I20240507 08:12:18.654498 6513 ts_tablet_manager.cc:1909] T b1fadbfd330b4e699e151f83125da281 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:18.652747 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.651098 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 962144051ca3423d9bb8527229dd1b94 (DEFAULT_TABLE table=default.Table_289cce254034416eb97d18b1a1e6868f [id=185f9e52ad81487ead9068bb3d3ab534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.655740 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962144051ca3423d9bb8527229dd1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.651932 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 61255088a80b4c448b1b048e5ada5bb9 (DEFAULT_TABLE table=default.Table_eb2b1328f64f4a5d8ebdc6d68d4282a9 [id=43d81fcebfe1457ea9190ec20bb94ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.659309 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61255088a80b4c448b1b048e5ada5bb9. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:18.659435 5654 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:18.659709 5654 leader_election.cc:341] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.667032 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa454f8e2b4c2386c63c3cf765b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.669003 5840 tablet_bootstrap.cc:492] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:18.669128 5840 ts_tablet_manager.cc:1367] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.000s
I20240507 08:12:18.669788 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f962ee766947fd83c7e016c1a901d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.670280 5840 raft_consensus.cc:348] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.670511 5840 raft_consensus.cc:374] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.670567 5840 raft_consensus.cc:729] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.670778 5840 consensus_queue.cc:260] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.674237 5840 ts_tablet_manager.cc:1398] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240507 08:12:18.674487 5840 tablet_bootstrap.cc:492] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:18.675838 5840 tablet_bootstrap.cc:654] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.678495 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1171) took 2484 ms. Trace:
I20240507 08:12:18.678591 7079 rpcz_store.cc:276] 0507 08:12:16.194328 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.231738 (+ 37410us) service_pool.cc:225] Handling call
0507 08:12:18.346104 (+2114366us) ts_tablet_manager.cc:1120] Deleting tablet 6b4c69d38e484f5c819b7dd915bf4410
0507 08:12:18.346114 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.434773 (+ 88659us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.603153 (+168380us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.678477 (+ 75324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2084887}
W20240507 08:12:18.678870 5654 consensus_peers.cc:485] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.679001 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 6b4c69d38e484f5c819b7dd915bf4410 (table default.Table_d73950d9f420413693022530757e7ead [id=fd965cac9b3647f3be3ddc94e8abf3d4]) successfully deleted
I20240507 08:12:18.685710 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1182) took 3163 ms. Trace:
I20240507 08:12:18.685873 6513 rpcz_store.cc:276] 0507 08:12:15.522489 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.522543 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:18.470694 (+2948151us) ts_tablet_manager.cc:1120] Deleting tablet b1fadbfd330b4e699e151f83125da281
0507 08:12:18.470702 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.594511 (+123809us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.653896 (+ 59385us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.685623 (+ 31727us) spinlock_profiling.cc:243] Waited 30.6 ms on lock 0x6160000b7154. stack: 00007f2d4efae756 00007f2d4efae510 00007f2d4dfe148d 0000000000955804 00000000009557cd 000000000095519a 00007f2d5a81c88a 00007f2d5a7dc1c4 00007f2d5baf89d6 00007f2d5e58a8d8 00007f2d5e5bc844 00007f2d5e59db30 00007f2d5e59d8bd 0000000000a60bcc 00007f2d4eff4b74 00007f2d4eff819b
0507 08:12:18.685693 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":70391680,"tablet-delete.queue_time_us":2947991}
I20240507 08:12:18.686411 6513 tablet_replica.cc:330] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:18.686658 6513 raft_consensus.cc:2229] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:18.687021 6513 raft_consensus.cc:2258] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.687644 6513 ts_tablet_manager.cc:1875] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.689358 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b1fadbfd330b4e699e151f83125da281 (table default.Table_307c9baefe434bb68c8d0906a9b9788e [id=6415c5a6c29747b6b005de15fb2ab365]) successfully deleted
I20240507 08:12:18.691962 5843 ts_tablet_manager.cc:1398] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.205s user 0.001s sys 0.001s
I20240507 08:12:18.695626 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1174) took 2333 ms. Trace:
I20240507 08:12:18.695724 7079 rpcz_store.cc:276] 0507 08:12:16.361791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.362394 (+ 603us) service_pool.cc:225] Handling call
0507 08:12:18.695493 (+2333099us) ts_tablet_manager.cc:1120] Deleting tablet 8bb62a72ed2248f6886823b9c64ff4c6
0507 08:12:18.695505 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.695605 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2332851}
W20240507 08:12:18.696136 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8bb62a72ed2248f6886823b9c64ff4c6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8bb62a72ed2248f6886823b9c64ff4c6 already in progress: creating tablet
I20240507 08:12:18.644843 5377 leader_election.cc:302] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.697556 5424 tablet_service.cc:1467] Processing CreateTablet for tablet 863c67d4ad97490eb6179ad65113bd4b (DEFAULT_TABLE table=default.Table_fca4a8f0d4e244c0872dd877f3e6aec8 [id=6d9d8819c44c41e0bfdb60f3f8ebd275]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:18.698869 5378 consensus_peers.cc:485] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:18.699811 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 80c892dab4414411acce4dd160079672 (table default.Table_25848992534e410cb5d2e01cb8287dc2 [id=59986073ff1148b8bf0693fb7749cdc3], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 80c892dab4414411acce4dd160079672 consensus state has no leader
I20240507 08:12:18.700165 5840 tablet_bootstrap.cc:492] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:18.700343 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863c67d4ad97490eb6179ad65113bd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.700579 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1173) took 2338 ms. Trace:
I20240507 08:12:18.700661 7079 rpcz_store.cc:276] 0507 08:12:16.361684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.363936 (+ 2252us) service_pool.cc:225] Handling call
0507 08:12:18.700474 (+2336538us) ts_tablet_manager.cc:1120] Deleting tablet b8c6f552832a45c4a2e28f40797db8fa
0507 08:12:18.700490 (+ 16us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.700563 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2336364}
I20240507 08:12:18.700779 5840 ts_tablet_manager.cc:1367] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.001s
W20240507 08:12:18.701125 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b8c6f552832a45c4a2e28f40797db8fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b8c6f552832a45c4a2e28f40797db8fa already in progress: creating tablet
I20240507 08:12:18.701443 5840 raft_consensus.cc:348] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.701740 5840 raft_consensus.cc:374] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.701853 5840 raft_consensus.cc:729] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:18.702418 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 57750.9us
I20240507 08:12:18.703727 5839 tablet_bootstrap.cc:492] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:18.703856 5839 ts_tablet_manager.cc:1367] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.272s user 0.003s sys 0.005s
I20240507 08:12:18.703914 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 2414f836bbcb4c8ebd1d3ac8320308d6 (DEFAULT_TABLE table=default.Table_118c82c5ecc644d998c3d38c16544c42 [id=e814c21c288e442a8157eca400936250]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.704005 5418 tablet_service.cc:1467] Processing CreateTablet for tablet dc13f0fb39074793b41e7b3b0bc595e4 (DEFAULT_TABLE table=default.Table_830c53deefa541df8e6682e8c3ef976f [id=f2ce4de6ab6647d4a9426aba3c43bdeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.704531 5839 raft_consensus.cc:348] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.704699 5839 raft_consensus.cc:374] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.704778 5839 raft_consensus.cc:729] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.704663 5840 consensus_queue.cc:260] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.704949 5839 consensus_queue.cc:260] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.707700 7552 raft_consensus.cc:2726] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.708129 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc13f0fb39074793b41e7b3b0bc595e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.708148 5840 ts_tablet_manager.cc:1398] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:18.708307 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 72753784f6ea4166affb413877903e58 (DEFAULT_TABLE table=default.Table_54786977c4974775aaa70909567f71fe [id=3d5211ad8e2349b9b7b2d76c195d68be]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.708475 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2414f836bbcb4c8ebd1d3ac8320308d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.708616 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 72753784f6ea4166affb413877903e58. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.709412 5413 tablet_service.cc:1467] Processing CreateTablet for tablet adfa454f8e2b4c2386c63c3cf765b5b1 (DEFAULT_TABLE table=default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46 [id=31cf7b29628549ab95a414a28a7e0e73]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.710543 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adfa454f8e2b4c2386c63c3cf765b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.711534 5425 tablet_service.cc:1467] Processing CreateTablet for tablet 6419b9be07564394a3b88ee36a14add3 (DEFAULT_TABLE table=default.Table_8fc4b266d274493fb54ebb06278ac446 [id=bad0680052bb41b48271e4330ccfd840]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.711833 5411 tablet_service.cc:1467] Processing CreateTablet for tablet 03069758ca5945aa9985aa9571ce2bfc (DEFAULT_TABLE table=default.Table_4d6a8373656e4d0da8b3529540c558af [id=71ac5228a8404365be05eda36fb60fac]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.712363 5420 tablet_service.cc:1467] Processing CreateTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 (DEFAULT_TABLE table=default.Table_9fcb6776bc164f07ba3ad8154d10bf36 [id=1dc69fa11458492cac07f67e31af8a80]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.712430 5839 ts_tablet_manager.cc:1398] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:18.712713 5839 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:18.713004 5410 tablet_service.cc:1467] Processing CreateTablet for tablet e5f962ee766947fd83c7e016c1a901d1 (DEFAULT_TABLE table=default.Table_19a5c269746d46d7b76ea68c04acd280 [id=0353eb24b6b04bba924a2b0f33f61ff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.713487 5843 tablet_bootstrap.cc:492] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:18.714075 5839 tablet_bootstrap.cc:654] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.714639 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f702440833504e4eb59747ee6de0cefb" } new_table_name: "default.Table_f7583788c28a4279a8cd5d8684eb4393" modify_external_catalogs: true
I20240507 08:12:18.714934 5425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6419b9be07564394a3b88ee36a14add3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.716117 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2e040bae1e56467b95bed23b37375389"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.716593 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e040bae1e56467b95bed23b37375389 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.716552 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 7ac3765fa3484f88ad6fd930d289fd7a (DEFAULT_TABLE table=default.Table_f3940a8999cb4633b1d82922022afe2f [id=58525b203eb34740a2cc94330eb01be4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.717926 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bd9f3573c06a41fcbf0fd31c87414464" } new_table_name: "default.Table_d853e82cbdae4ea3b215934ff843d3ef" modify_external_catalogs: true
I20240507 08:12:18.718106 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9211028e9eeb452d8ab5903710349ca6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.718595 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 401402afc04f4a9082c02ca5bf39d6a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.719635 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ebd7060e386f4e7cae3ae4ec34a2fa30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.720060 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebd7060e386f4e7cae3ae4ec34a2fa30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.720199 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ac3765fa3484f88ad6fd930d289fd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.720541 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5f962ee766947fd83c7e016c1a901d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.721148 5426 tablet_service.cc:1467] Processing CreateTablet for tablet 7202cad072874dfcad0a9ad563a0bde4 (DEFAULT_TABLE table=default.Table_ec6c5b0ac0274d1fb210bcf62f46df8d [id=452fa515268b42cebabfcd5f7573b4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.716897 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 4d99f656e19349c1935294fd7968ecd1 (DEFAULT_TABLE table=default.Table_b5153ea0c2884e12a0fbd2cb2fb5678a [id=b21deda92f9544ecada5c6bddc42b973]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.723491 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5aca2077b76b49e3bf9daae57dc09b8f
I20240507 08:12:18.724699 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ffc89e00efaa4bde9242be40cf409c59
I20240507 08:12:18.726619 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 3f519b1ee222487a94e1625c6bb7c181 (DEFAULT_TABLE table=default.Table_cc67c2a61fe048a0acc93d29961dcdce [id=319412836ddf40908b62a2eab165511e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.727135 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 120bce4b20714fc1a0e1237abc00d5bb
I20240507 08:12:18.729841 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc89e00efaa4bde9242be40cf409c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.734503 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc89e00efaa4bde9242be40cf409c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.734933 5840 tablet_bootstrap.cc:492] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:18.735039 5426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7202cad072874dfcad0a9ad563a0bde4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.735265 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03069758ca5945aa9985aa9571ce2bfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.735455 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f519b1ee222487a94e1625c6bb7c181. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.735769 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d99f656e19349c1935294fd7968ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.729676 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ed7d7d2e8e424c90048c87c0fb0ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.736441 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.736497 5445 raft_consensus.cc:2454] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:18.736943 5840 tablet_bootstrap.cc:654] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.739437 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.741551 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.742746 5843 tablet_bootstrap.cc:654] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.704707 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1178) took 2158 ms. Trace:
I20240507 08:12:18.748740 7079 rpcz_store.cc:276] 0507 08:12:16.546125 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.553644 (+ 7519us) service_pool.cc:225] Handling call
0507 08:12:18.700902 (+2147258us) ts_tablet_manager.cc:1120] Deleting tablet 310c601abead42e0bd2a2df1a0e16ec2
0507 08:12:18.700910 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.704689 (+ 3779us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2147052}
W20240507 08:12:18.749390 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 310c601abead42e0bd2a2df1a0e16ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310c601abead42e0bd2a2df1a0e16ec2 already in progress: creating tablet
I20240507 08:12:18.754835 5839 tablet_bootstrap.cc:492] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:18.755100 5839 ts_tablet_manager.cc:1367] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20240507 08:12:18.755226 5841 ts_tablet_manager.cc:1888] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.755329 5841 log.cc:1196] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/14929efb0ed049d6b554598a5b8cb494
I20240507 08:12:18.755692 5841 ts_tablet_manager.cc:1909] T 14929efb0ed049d6b554598a5b8cb494 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.756039 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1169) took 3078 ms. Trace:
I20240507 08:12:18.756325 5839 raft_consensus.cc:348] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.756181 5841 rpcz_store.cc:276] 0507 08:12:15.677827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.678565 (+ 738us) service_pool.cc:225] Handling call
0507 08:12:18.411404 (+2732839us) ts_tablet_manager.cc:1120] Deleting tablet 14929efb0ed049d6b554598a5b8cb494
0507 08:12:18.411415 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.481549 (+ 70134us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.755129 (+273580us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.756020 (+ 891us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2715683}
I20240507 08:12:18.756582 5839 raft_consensus.cc:374] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.756678 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 61255088a80b4c448b1b048e5ada5bb9 (DEFAULT_TABLE table=default.Table_eb2b1328f64f4a5d8ebdc6d68d4282a9 [id=43d81fcebfe1457ea9190ec20bb94ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.756850 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 14929efb0ed049d6b554598a5b8cb494 (table default.Table_b21b58e8d49943268e7491117d5d717e [id=afe7e0cf53aa444f84f4773519b48c85]) successfully deleted
I20240507 08:12:18.757076 5841 tablet_replica.cc:330] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.757282 5841 raft_consensus.cc:2229] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.757309 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 962144051ca3423d9bb8527229dd1b94 (DEFAULT_TABLE table=default.Table_289cce254034416eb97d18b1a1e6868f [id=185f9e52ad81487ead9068bb3d3ab534]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.757688 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61255088a80b4c448b1b048e5ada5bb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.757436 5841 raft_consensus.cc:2258] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.758296 5839 raft_consensus.cc:729] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.758481 5841 ts_tablet_manager.cc:1875] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.758555 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 962144051ca3423d9bb8527229dd1b94. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.758754 5582 raft_consensus.cc:3037] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.758653 5839 consensus_queue.cc:260] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.766923 5839 ts_tablet_manager.cc:1398] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240507 08:12:18.771083 7079 tablet_replica.cc:330] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:18.771309 7079 raft_consensus.cc:2229] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.771540 7079 raft_consensus.cc:2258] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:18.772701 5520 leader_election.cc:341] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:12:18.772830 5520 leader_election.cc:302] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:18.772889 7079 ts_tablet_manager.cc:1875] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.773217 5520 leader_election.cc:302] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:18.773532 7602 raft_consensus.cc:2726] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.773499 5656 leader_election.cc:302] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:18.773888 5656 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f2de26190bc44916809856d84ad68345
I20240507 08:12:18.774001 5656 leader_election.cc:302] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.774717 7100 consensus_queue.cc:1035] T 2a77151b173d448aa247317ec6160ce0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.775034 5839 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:18.775187 7100 raft_consensus.cc:2781] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:18.775267 7100 raft_consensus.cc:482] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:18.775357 7100 raft_consensus.cc:3037] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.776015 6675 raft_consensus.cc:2726] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.776326 6675 consensus_queue.cc:1035] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.782171 6513 ts_tablet_manager.cc:1888] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.783008 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.784895 6513 log.cc:1196] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c707458d1c094d87b2204683b28b49e4
I20240507 08:12:18.785673 5417 tablet_service.cc:1467] Processing CreateTablet for tablet e54273ed92f640d98b853a7661ae3e26 (DEFAULT_TABLE table=default.Table_bac149077ee54ee3ba57739c3b704a42 [id=362c05703392453ebeef62e6937a9139]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.789458 7100 raft_consensus.cc:504] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.794903 5839 tablet_bootstrap.cc:654] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.794937 7100 leader_election.cc:288] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.799273 5419 tablet_service.cc:1467] Processing CreateTablet for tablet c74576ac62e04fb7a732f54e7b82e0ef (DEFAULT_TABLE table=default.Table_010a0902019b4e9381337fc5cd6a3834 [id=a46a6abbe82d4f078ddc784624b0218b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:18.799697 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c74576ac62e04fb7a732f54e7b82e0ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.800206 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e54273ed92f640d98b853a7661ae3e26. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:18.803059 7602 raft_consensus.cc:2726] T 9211028e9eeb452d8ab5903710349ca6 P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.805274 7550 consensus_queue.cc:1035] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:18.807427 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ed7d7d2e8e424c90048c87c0fb0ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:18.811252 5654 leader_election.cc:341] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.838078 6513 ts_tablet_manager.cc:1909] T c707458d1c094d87b2204683b28b49e4 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:18.839233 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1177) took 3324 ms. Trace:
I20240507 08:12:18.839475 6513 rpcz_store.cc:276] 0507 08:12:15.514375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.514883 (+ 508us) service_pool.cc:225] Handling call
0507 08:12:18.686369 (+3171486us) ts_tablet_manager.cc:1120] Deleting tablet c707458d1c094d87b2204683b28b49e4
0507 08:12:18.686377 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.693792 (+ 7415us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.782063 (+ 88271us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.839205 (+ 57142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3154402}
I20240507 08:12:18.840005 6513 tablet_replica.cc:330] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:18.840394 6513 raft_consensus.cc:2229] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.840642 6513 raft_consensus.cc:2258] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.841668 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c707458d1c094d87b2204683b28b49e4 (table default.Table_6b76dffef8004978b959721044683956 [id=4f0c80f22ec84a1eabc46a98c8568a07]) successfully deleted
I20240507 08:12:18.845845 7079 ts_tablet_manager.cc:1888] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:18.846534 7650 tablet.cc:1722] T 8e6454c4a59d4c4cae1681bcae0da2d7 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:18.847087 7079 log.cc:1196] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/08aac6e88b7c41d8a59d1e12e7f6c14e
I20240507 08:12:18.849905 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_896f00e31db14579b52336dddabe10ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.850476 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_896f00e31db14579b52336dddabe10ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.851058 5841 ts_tablet_manager.cc:1888] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.851157 5841 log.cc:1196] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c839c8a5041f4c3da9bc0749afee1413
I20240507 08:12:18.851495 5841 ts_tablet_manager.cc:1909] T c839c8a5041f4c3da9bc0749afee1413 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.851195 7079 ts_tablet_manager.cc:1909] T 08aac6e88b7c41d8a59d1e12e7f6c14e P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:18.852792 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a14accce1d384c4b8f7772785c405082"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.853243 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a14accce1d384c4b8f7772785c405082 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.857508 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 04a14cf04d7a4ce3a2df939d0db515b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.857743 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aca2077b76b49e3bf9daae57dc09b8f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.857901 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc89e00efaa4bde9242be40cf409c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.858028 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 120bce4b20714fc1a0e1237abc00d5bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.858907 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet fdffec6ce0da4c409466783bd76a6b4b
I20240507 08:12:18.859581 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bec6672c91654fe68356ce93a8ea725a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.860031 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bec6672c91654fe68356ce93a8ea725a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.860134 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet fdffec6ce0da4c409466783bd76a6b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.865000 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet b27cc38d0c4c4083900cd6765a5cf6e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.865440 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1180) took 2310 ms. Trace:
I20240507 08:12:18.866345 7079 rpcz_store.cc:276] 0507 08:12:16.554921 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.565368 (+ 10447us) service_pool.cc:225] Handling call
0507 08:12:18.749847 (+2184479us) ts_tablet_manager.cc:1120] Deleting tablet 08aac6e88b7c41d8a59d1e12e7f6c14e
0507 08:12:18.771021 (+ 21174us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.796267 (+ 25246us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.845023 (+ 48756us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.865420 (+ 20397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2184192}
I20240507 08:12:18.869278 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]) successfully deleted
I20240507 08:12:18.872942 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 957775c01be94cc081db344c95a81310 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:18.873680 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet fdffec6ce0da4c409466783bd76a6b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.873936 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def4bf6a7554a4090a39b1823c4097c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:18.874496 7079 tablet_replica.cc:330] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:18.875983 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42809e2f45e9466594377b41ae920f3b consensus state has no leader
I20240507 08:12:18.876505 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_db5191b1f46846118fc8702ca6d48fe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.877125 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db5191b1f46846118fc8702ca6d48fe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.876922 7079 raft_consensus.cc:2229] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.880369 7079 raft_consensus.cc:2258] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.881197 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4892af21ff1c43deb37b30758ce5ec7c" } new_table_name: "default.Table_dda39a08ef52408e8400599afc72c809" modify_external_catalogs: true
I20240507 08:12:18.884387 7079 ts_tablet_manager.cc:1875] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.888504 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1170) took 3210 ms. Trace:
I20240507 08:12:18.888787 5841 rpcz_store.cc:276] 0507 08:12:15.677928 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.678724 (+ 796us) service_pool.cc:225] Handling call
0507 08:12:18.757023 (+3078299us) ts_tablet_manager.cc:1120] Deleting tablet c839c8a5041f4c3da9bc0749afee1413
0507 08:12:18.757032 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.768521 (+ 11489us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.850978 (+ 82457us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.888482 (+ 37504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3060967}
I20240507 08:12:18.889250 5841 tablet_replica.cc:330] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.889264 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c839c8a5041f4c3da9bc0749afee1413 (table default.Table_e19ac67deb044b61b0e6194e8b54aae0 [id=9c2cb8887f5344ff8a6d58b3f7a72459]) successfully deleted
I20240507 08:12:18.889605 5841 raft_consensus.cc:2229] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.889827 5841 raft_consensus.cc:2258] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.890635 5841 ts_tablet_manager.cc:1875] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:18.890614 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 116054us
W20240507 08:12:18.891029 5656 consensus_peers.cc:485] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.896960 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3ed7d7d2e8e424c90048c87c0fb0ca6" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:18.897186 5442 raft_consensus.cc:3037] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.900928 5442 raft_consensus.cc:2454] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:18.903131 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet fdffec6ce0da4c409466783bd76a6b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:18 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:18.904735 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.120s user 0.001s sys 0.000s
I20240507 08:12:18.909430 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet b60ec15a83934999871425ef9a64ec72 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:18.922180 5839 tablet_bootstrap.cc:492] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:18.922405 5839 ts_tablet_manager.cc:1367] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.008s
I20240507 08:12:18.923056 5839 raft_consensus.cc:348] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.923326 5839 raft_consensus.cc:374] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.923431 5839 raft_consensus.cc:729] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.923743 5839 consensus_queue.cc:260] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:18.925423 7649 raft_consensus.cc:482] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.925585 7649 raft_consensus.cc:504] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.926240 7649 leader_election.cc:288] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:18.927280 5654 consensus_peers.cc:485] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:18.927561 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:18.928072 5843 tablet_bootstrap.cc:492] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:18.928244 5843 ts_tablet_manager.cc:1367] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.228s user 0.007s sys 0.003s
W20240507 08:12:18.928236 5654 leader_election.cc:341] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.929122 7358 raft_consensus.cc:482] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.929147 5656 leader_election.cc:302] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:18.929235 7358 raft_consensus.cc:504] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.929450 5843 raft_consensus.cc:348] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.929662 5843 raft_consensus.cc:374] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:18.929714 5843 raft_consensus.cc:729] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:18.929752 7358 leader_election.cc:288] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.929837 7579 raft_consensus.cc:482] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.929894 5843 consensus_queue.cc:260] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.929925 7579 raft_consensus.cc:504] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.930099 7358 raft_consensus.cc:2781] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:18.930182 7358 raft_consensus.cc:686] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:18.930367 7579 leader_election.cc:288] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:18.930487 5843 ts_tablet_manager.cc:1398] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240507 08:12:18.930428 7358 consensus_queue.cc:237] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.930886 5843 tablet_bootstrap.cc:492] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:18.932554 5839 ts_tablet_manager.cc:1398] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:12:18.932804 5839 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:18.934597 5839 tablet_bootstrap.cc:654] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.942441 5843 tablet_bootstrap.cc:654] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:18.948941 6513 ts_tablet_manager.cc:1875] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.952070 5841 ts_tablet_manager.cc:1888] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:18.952169 5841 log.cc:1196] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4e5439844f384513b4719cfe4f4e5d9b
I20240507 08:12:18.953604 5841 ts_tablet_manager.cc:1909] T 4e5439844f384513b4719cfe4f4e5d9b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.954216 7079 ts_tablet_manager.cc:1888] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:18.954368 7079 log.cc:1196] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/43644b4405f44bfd9a1afcaedbd68db5
I20240507 08:12:18.954667 7079 ts_tablet_manager.cc:1909] T 43644b4405f44bfd9a1afcaedbd68db5 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:18.955596 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1175) took 2452 ms. Trace:
I20240507 08:12:18.957190 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1171) took 3279 ms. Trace:
I20240507 08:12:18.957695 7079 rpcz_store.cc:276] 0507 08:12:16.502934 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.608778 (+105844us) service_pool.cc:225] Handling call
0507 08:12:18.872669 (+2263891us) ts_tablet_manager.cc:1120] Deleting tablet 43644b4405f44bfd9a1afcaedbd68db5
0507 08:12:18.872679 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.915532 (+ 42853us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.954121 (+ 38589us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.955574 (+ 1453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263664}
I20240507 08:12:18.957294 5841 rpcz_store.cc:276] 0507 08:12:15.678016 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.678825 (+ 809us) service_pool.cc:225] Handling call
0507 08:12:18.889162 (+3210337us) ts_tablet_manager.cc:1120] Deleting tablet 4e5439844f384513b4719cfe4f4e5d9b
0507 08:12:18.889170 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.913929 (+ 24759us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.951981 (+ 38052us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.957171 (+ 5190us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3193317}
I20240507 08:12:18.960868 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4e5439844f384513b4719cfe4f4e5d9b (table default.Table_9342125a817c439eb044fbe82437c21a [id=64f54c3a3fba416f8810335c6f651ced]) successfully deleted
I20240507 08:12:18.960932 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]) successfully deleted
I20240507 08:12:18.961386 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1172) took 3283 ms. Trace:
I20240507 08:12:18.961467 5841 rpcz_store.cc:276] 0507 08:12:15.678096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.678911 (+ 815us) service_pool.cc:225] Handling call
0507 08:12:18.961267 (+3282356us) ts_tablet_manager.cc:1120] Deleting tablet 5471c74c33cd42e8aaf47f71f61adff7
0507 08:12:18.961278 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.961369 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265380}
W20240507 08:12:18.961838 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5471c74c33cd42e8aaf47f71f61adff7 because the tablet was not found. No further retry: Not found: Tablet not found: 5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:18.963879 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.965179 5841 tablet_replica.cc:330] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:18.965605 5841 raft_consensus.cc:2229] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:18.967217 5841 raft_consensus.cc:2258] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.968974 5841 ts_tablet_manager.cc:1875] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:18.971421 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb68d3d3d5c842b0a95906db237261d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.971946 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb68d3d3d5c842b0a95906db237261d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.973186 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 2e44c3f6f96e417b92878d8ba4ff88a7 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:18.973814 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a3550d2ca3f04d659cd01f9eb838edc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.974278 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3550d2ca3f04d659cd01f9eb838edc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:18.975579 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 19e8ef286cf247278fe2b01218e6310a consensus state has no leader
I20240507 08:12:18.976435 7552 raft_consensus.cc:482] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:18.976560 7552 raft_consensus.cc:504] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:18.977249 7552 leader_election.cc:288] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:18.978000 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:18.978489 5378 leader_election.cc:341] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.977562 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6362b6a3832a42099cbf4fc493c068fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.980033 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6362b6a3832a42099cbf4fc493c068fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:18.980304 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1185) took 2324 ms. Trace:
I20240507 08:12:18.980394 7079 rpcz_store.cc:276] 0507 08:12:16.655812 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.656571 (+ 759us) service_pool.cc:225] Handling call
0507 08:12:18.980167 (+2323596us) ts_tablet_manager.cc:1120] Deleting tablet 994f50ea19a2445288663b7d7b7ad291
0507 08:12:18.980178 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.980278 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2323254}
I20240507 08:12:18.980605 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_344c146019d84e2f8e80331a5530bccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:18.980825 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 994f50ea19a2445288663b7d7b7ad291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994f50ea19a2445288663b7d7b7ad291 already in progress: creating tablet
I20240507 08:12:18.977995 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_05652d1f4f574b078ce9726ea093c0eb" } new_table_name: "default.Table_e89359ce687c43dca271bf6f0bc015b1" modify_external_catalogs: true
W20240507 08:12:18.980993 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_344c146019d84e2f8e80331a5530bccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:18.985749 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 56233.5us
I20240507 08:12:18.985822 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1184) took 2331 ms. Trace:
I20240507 08:12:18.985908 7079 rpcz_store.cc:276] 0507 08:12:16.654113 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.657622 (+ 3509us) service_pool.cc:225] Handling call
0507 08:12:18.985699 (+2328077us) ts_tablet_manager.cc:1120] Deleting tablet 9669c1169b5244218383f7aa7fbc3b85
0507 08:12:18.985709 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.985801 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2327266}
I20240507 08:12:18.986757 5841 ts_tablet_manager.cc:1888] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240507 08:12:18.986965 5656 leader_election.cc:341] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.987121 5656 leader_election.cc:302] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:18.987185 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 9669c1169b5244218383f7aa7fbc3b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9669c1169b5244218383f7aa7fbc3b85 already in progress: creating tablet
I20240507 08:12:18.987346 6675 raft_consensus.cc:2726] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:18.987617 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1183) took 2333 ms. Trace:
I20240507 08:12:18.987983 5841 log.cc:1196] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/99e4514039304f0696c69a057db0ea6e
I20240507 08:12:18.988116 7079 rpcz_store.cc:276] 0507 08:12:16.653987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.660048 (+ 6061us) service_pool.cc:225] Handling call
0507 08:12:18.987507 (+2327459us) ts_tablet_manager.cc:1120] Deleting tablet 73f82413951c48a89e7db8fe9e50c749
0507 08:12:18.987518 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.987601 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2326628}
W20240507 08:12:18.988987 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 73f82413951c48a89e7db8fe9e50c749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f82413951c48a89e7db8fe9e50c749 already in progress: creating tablet
I20240507 08:12:18.989720 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.990599 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1191) took 2252 ms. Trace:
I20240507 08:12:18.991205 5841 ts_tablet_manager.cc:1909] T 99e4514039304f0696c69a057db0ea6e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:18.991252 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:18.991701 5442 raft_consensus.cc:2454] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:18.991325 7079 rpcz_store.cc:276] 0507 08:12:16.738172 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.740977 (+ 2805us) service_pool.cc:225] Handling call
0507 08:12:18.990500 (+2249523us) ts_tablet_manager.cc:1120] Deleting tablet 17c6f752a0d4485e9606a5570a0d6c96
0507 08:12:18.990509 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.990582 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2247408}
W20240507 08:12:18.992501 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 17c6f752a0d4485e9606a5570a0d6c96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17c6f752a0d4485e9606a5570a0d6c96 already in progress: creating tablet
I20240507 08:12:18.992985 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1173) took 3314 ms. Trace:
I20240507 08:12:18.993276 7079 tablet_replica.cc:330] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:18.993782 7079 raft_consensus.cc:2229] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:18.998026 5656 consensus_peers.cc:485] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 975f22194aff40609a3cfa63b9b3aca2. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:18.998265 5656 leader_election.cc:341] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:18.998539 5656 leader_election.cc:302] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:18.998092 5841 rpcz_store.cc:276] 0507 08:12:15.678178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.679012 (+ 834us) service_pool.cc:225] Handling call
0507 08:12:18.965132 (+3286120us) ts_tablet_manager.cc:1120] Deleting tablet 99e4514039304f0696c69a057db0ea6e
0507 08:12:18.965142 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.972672 (+ 7530us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.986640 (+ 13968us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.992963 (+ 6323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269213}
I20240507 08:12:18.998827 7100 raft_consensus.cc:2781] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:18.998942 7079 raft_consensus.cc:2258] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:18.999356 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 99e4514039304f0696c69a057db0ea6e (table default.Table_fc64369fe72742ccb59a630ee95a8007 [id=54ceed7b5cad41ecbacc40ace9f48cd5]) successfully deleted
I20240507 08:12:18.998957 7100 raft_consensus.cc:482] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:18.999715 7100 raft_consensus.cc:3037] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:18.999943 7579 consensus_queue.cc:1035] T 66cd3dbd84234e398a6151483d447260 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.000128 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1174) took 3321 ms. Trace:
I20240507 08:12:19.000913 7079 ts_tablet_manager.cc:1875] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.001222 5841 rpcz_store.cc:276] 0507 08:12:15.678279 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.679085 (+ 806us) service_pool.cc:225] Handling call
0507 08:12:19.000028 (+3320943us) ts_tablet_manager.cc:1120] Deleting tablet cd4379b6fded4a52843f55f27ae67121
0507 08:12:19.000038 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.000112 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3304027}
W20240507 08:12:19.001971 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cd4379b6fded4a52843f55f27ae67121 because the tablet was not found. No further retry: Not found: Tablet not found: cd4379b6fded4a52843f55f27ae67121
I20240507 08:12:19.002467 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1175) took 3270 ms. Trace:
I20240507 08:12:19.002499 7100 raft_consensus.cc:504] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.004647 5841 rpcz_store.cc:276] 0507 08:12:15.732364 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.732428 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:19.002343 (+3269915us) ts_tablet_manager.cc:1120] Deleting tablet ddcb720c09c5486995391837fc54e7ce
0507 08:12:19.002354 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.002453 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3252595}
W20240507 08:12:19.005718 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ddcb720c09c5486995391837fc54e7ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb720c09c5486995391837fc54e7ce already in progress: creating tablet
I20240507 08:12:19.007022 7100 leader_election.cc:288] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.009896 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1176) took 3277 ms. Trace:
I20240507 08:12:19.013852 5840 tablet_bootstrap.cc:492] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.014027 5840 ts_tablet_manager.cc:1367] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.279s user 0.062s sys 0.083s
I20240507 08:12:19.014703 5840 raft_consensus.cc:348] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.014968 5840 raft_consensus.cc:374] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.015051 5840 raft_consensus.cc:729] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.015789 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2c3d31f3a01b41d59815566c10c96fe8" } modify_external_catalogs: true
I20240507 08:12:19.016006 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2c3d31f3a01b41d59815566c10c96fe8" } modify_external_catalogs: true
I20240507 08:12:19.016817 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f218240b99734c72a1ef75b6e7bb6534" } modify_external_catalogs: true
I20240507 08:12:19.016969 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f218240b99734c72a1ef75b6e7bb6534" } modify_external_catalogs: true
I20240507 08:12:19.017064 5840 consensus_queue.cc:260] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.017478 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.018224 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.018440 5840 ts_tablet_manager.cc:1398] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240507 08:12:19.018616 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d55b33508cc4645b468556b1384ffda" } modify_external_catalogs: true
I20240507 08:12:19.018750 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1d55b33508cc4645b468556b1384ffda" } modify_external_catalogs: true
I20240507 08:12:19.019541 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e57fcc9d421a4abdb75cb707e06febbe" } modify_external_catalogs: true
I20240507 08:12:19.019675 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e57fcc9d421a4abdb75cb707e06febbe" } modify_external_catalogs: true
W20240507 08:12:19.021308 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f3b373f17dc40efa05dd024318393c5 consensus state has no leader
I20240507 08:12:19.023928 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:19.024835 5840 tablet_bootstrap.cc:492] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.027724 5840 tablet_bootstrap.cc:654] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:19.034137 5654 leader_election.cc:341] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.034484 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:19.010936 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47a154769d0f4501837888feba98f33d" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:19.036856 5446 raft_consensus.cc:3037] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:19.038467 5654 leader_election.cc:341] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.038599 5654 leader_election.cc:302] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.039503 7100 raft_consensus.cc:2726] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.040120 5446 raft_consensus.cc:2454] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:19.040617 5656 leader_election.cc:302] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:19.041023 7649 raft_consensus.cc:2781] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:19.040961 5841 rpcz_store.cc:276] 0507 08:12:15.732827 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.732902 (+ 75us) service_pool.cc:225] Handling call
0507 08:12:19.009734 (+3276832us) ts_tablet_manager.cc:1120] Deleting tablet f9b60e7fd01c4c96911fff6bf62ade96
0507 08:12:19.009749 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.009877 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259926}
I20240507 08:12:19.041182 7649 raft_consensus.cc:686] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:19.041497 7649 consensus_queue.cc:237] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:19.041952 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f9b60e7fd01c4c96911fff6bf62ade96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b60e7fd01c4c96911fff6bf62ade96 already in progress: creating tablet
I20240507 08:12:19.043095 7663 tablet.cc:1722] T 8e6454c4a59d4c4cae1681bcae0da2d7 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:19.043457 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1183) took 3224 ms. Trace:
I20240507 08:12:19.043736 5841 rpcz_store.cc:276] 0507 08:12:15.819094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.819434 (+ 340us) service_pool.cc:225] Handling call
0507 08:12:19.042673 (+3223239us) ts_tablet_manager.cc:1120] Deleting tablet 38fc0180e9d34d11885b819529407930
0507 08:12:19.042684 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.043439 (+ 755us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3223014}
I20240507 08:12:19.045626 5840 tablet_bootstrap.cc:492] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.045924 5840 ts_tablet_manager.cc:1367] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
W20240507 08:12:19.047035 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 38fc0180e9d34d11885b819529407930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38fc0180e9d34d11885b819529407930 already in progress: creating tablet
I20240507 08:12:19.046985 5840 raft_consensus.cc:348] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.047226 5840 raft_consensus.cc:374] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.047371 5840 raft_consensus.cc:729] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.053989 5843 tablet_bootstrap.cc:492] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.054183 5843 ts_tablet_manager.cc:1367] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.123s user 0.011s sys 0.000s
I20240507 08:12:19.055266 5839 tablet_bootstrap.cc:492] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.055408 5839 ts_tablet_manager.cc:1367] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.007s
I20240507 08:12:19.056025 5839 raft_consensus.cc:348] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.056253 5839 raft_consensus.cc:374] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.056318 5839 raft_consensus.cc:729] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.056560 5839 consensus_queue.cc:260] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.057324 5839 ts_tablet_manager.cc:1398] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20240507 08:12:19.057718 5654 leader_election.cc:341] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:19.064905 5377 leader_election.cc:341] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.065033 5377 leader_election.cc:302] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.065390 7552 raft_consensus.cc:2726] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.073858 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ec50c8f7242944919aadbff5d3b2a54a" } modify_external_catalogs: true
I20240507 08:12:19.074123 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ec50c8f7242944919aadbff5d3b2a54a" } modify_external_catalogs: true
I20240507 08:12:19.077123 7579 consensus_queue.cc:1035] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.080583 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_29d81546e07145b6aad46145e0f37793" } new_table_name: "default.Table_e5793f8bd4d74a78aa4c896307632ceb" modify_external_catalogs: true
W20240507 08:12:19.081689 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b1c1a5919b5247c8aa542b806611dfe7 consensus state has no leader
I20240507 08:12:19.082892 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 89db2847c09b4c74b809bd13f78d0b8b
I20240507 08:12:19.083775 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.084259 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.086428 5843 raft_consensus.cc:348] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.086709 5843 raft_consensus.cc:374] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.086817 5843 raft_consensus.cc:729] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.087095 5843 consensus_queue.cc:260] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.088670 7650 tablet.cc:1722] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20240507 08:12:18.997501 6513 ts_tablet_manager.cc:1888] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.089773 6513 log.cc:1196] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/055a30bfc61b4ff2be9f2418ca59b030
I20240507 08:12:19.090327 6513 ts_tablet_manager.cc:1909] T 055a30bfc61b4ff2be9f2418ca59b030 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.091389 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1183) took 3568 ms. Trace:
I20240507 08:12:19.091575 6513 rpcz_store.cc:276] 0507 08:12:15.522825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.534181 (+ 11356us) service_pool.cc:225] Handling call
0507 08:12:18.839945 (+3305764us) ts_tablet_manager.cc:1120] Deleting tablet 055a30bfc61b4ff2be9f2418ca59b030
0507 08:12:18.839958 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:18.993767 (+153809us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:18.997387 (+ 3620us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.091363 (+ 93976us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3305581}
I20240507 08:12:19.092321 6513 tablet_replica.cc:330] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.092629 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1209) took 2916 ms. Trace:
I20240507 08:12:19.092926 6513 raft_consensus.cc:2229] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:19.093336 6513 raft_consensus.cc:2258] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.094030 6513 ts_tablet_manager.cc:1875] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.096002 5839 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:12:19.099640 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8b7dcf5643fd4c18ba7a69011421b5b8 consensus state has no leader
I20240507 08:12:19.102450 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 055a30bfc61b4ff2be9f2418ca59b030 (table default.Table_97792a6d740f4599b202087719f29978 [id=afa10dc71a6b478a8de83111839425bc]) successfully deleted
I20240507 08:12:19.105278 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:19.111272 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5471c74c33cd42e8aaf47f71f61adff7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:54 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.113684 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet aa094773f3bf418492bcd5b5021fd429 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:19.117054 7079 ts_tablet_manager.cc:1888] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.117170 7079 log.cc:1196] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/31251576beee4a8a81a453b721c9c058
I20240507 08:12:19.117441 7079 ts_tablet_manager.cc:1909] T 31251576beee4a8a81a453b721c9c058 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:19.102481 5654 consensus_peers.cc:485] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.120095 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 89db2847c09b4c74b809bd13f78d0b8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.127161 5841 rpcz_store.cc:276] 0507 08:12:16.176068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.180143 (+ 4075us) service_pool.cc:225] Handling call
0507 08:12:19.047438 (+2867295us) ts_tablet_manager.cc:1120] Deleting tablet c20c60fe0aad4cb3b12d04ad1a87d9f4
0507 08:12:19.072045 (+ 24607us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.092609 (+ 20564us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2867121}
W20240507 08:12:19.127758 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 already in progress: creating tablet
I20240507 08:12:19.127877 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1207) took 2952 ms. Trace:
I20240507 08:12:19.127949 5841 rpcz_store.cc:276] 0507 08:12:16.175861 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.175934 (+ 73us) service_pool.cc:225] Handling call
0507 08:12:19.127744 (+2951810us) ts_tablet_manager.cc:1120] Deleting tablet 0ddb7dd28f43488581d3a75bf1d68a8a
0507 08:12:19.127754 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.127859 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2916342}
W20240507 08:12:19.128305 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 0ddb7dd28f43488581d3a75bf1d68a8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddb7dd28f43488581d3a75bf1d68a8a already in progress: creating tablet
I20240507 08:12:19.136442 5839 tablet_bootstrap.cc:654] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.142306 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 01820ecad55a4097bc5917a8f7949fe7
I20240507 08:12:19.142632 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:19.143410 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.144184 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1192) took 2405 ms. Trace:
I20240507 08:12:19.144819 5841 tablet_replica.cc:330] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:19.144973 7079 rpcz_store.cc:276] 0507 08:12:16.738309 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.741130 (+ 2821us) service_pool.cc:225] Handling call
0507 08:12:18.993222 (+2252092us) ts_tablet_manager.cc:1120] Deleting tablet 31251576beee4a8a81a453b721c9c058
0507 08:12:18.993234 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.052162 (+ 58928us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.116937 (+ 64775us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.143936 (+ 26999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2249930}
I20240507 08:12:19.146943 5841 raft_consensus.cc:2229] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:19.147864 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 31251576beee4a8a81a453b721c9c058 (table default.Table_ac127de16ef340a4a2c4ac27753d230d [id=1baf6ec4044743b1898671f2a5de9372]) successfully deleted
I20240507 08:12:19.148425 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1193) took 2364 ms. Trace:
I20240507 08:12:19.148707 5261 catalog_manager.cc:5484] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:19.149219 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
I20240507 08:12:19.146214 5843 ts_tablet_manager.cc:1398] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240507 08:12:19.149966 5841 raft_consensus.cc:2258] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.149109 7079 rpcz_store.cc:276] 0507 08:12:16.783647 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.783866 (+ 219us) service_pool.cc:225] Handling call
0507 08:12:19.147700 (+2363834us) ts_tablet_manager.cc:1120] Deleting tablet bee9e434083f4fa6859cfe2b7cca16e5
0507 08:12:19.147885 (+ 185us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.148405 (+ 520us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2362873}
W20240507 08:12:19.151732 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet bee9e434083f4fa6859cfe2b7cca16e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bee9e434083f4fa6859cfe2b7cca16e5 already in progress: creating tablet
I20240507 08:12:19.152419 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1197) took 2335 ms. Trace:
I20240507 08:12:19.153358 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet a905e345412541558b30c629a2eb86c5
I20240507 08:12:19.105559 5840 consensus_queue.cc:260] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.153893 5840 ts_tablet_manager.cc:1398] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.108s user 0.002s sys 0.000s
I20240507 08:12:19.155674 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.153427 7079 rpcz_store.cc:276] 0507 08:12:16.816781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.816835 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:19.151111 (+2334276us) ts_tablet_manager.cc:1120] Deleting tablet a68badb3b5644ed8b007b2b04bc4f6e1
0507 08:12:19.151125 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.152090 (+ 965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2333842}
I20240507 08:12:19.156508 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet a905e345412541558b30c629a2eb86c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:19.158869 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a68badb3b5644ed8b007b2b04bc4f6e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68badb3b5644ed8b007b2b04bc4f6e1 already in progress: creating tablet
I20240507 08:12:19.158785 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1199) took 2276 ms. Trace:
I20240507 08:12:19.160010 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet d3f120d9fbf542d39508903883ecb5e6 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:19.160640 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet a905e345412541558b30c629a2eb86c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.159837 7079 rpcz_store.cc:276] 0507 08:12:16.882183 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.890071 (+ 7888us) service_pool.cc:225] Handling call
0507 08:12:19.157700 (+2267629us) ts_tablet_manager.cc:1120] Deleting tablet 219bad2b972c45a8930d6f47ca361858
0507 08:12:19.157941 (+ 241us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.158766 (+ 825us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266971}
I20240507 08:12:19.161806 5840 tablet_bootstrap.cc:492] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.162940 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1201) took 2260 ms. Trace:
W20240507 08:12:19.163309 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 219bad2b972c45a8930d6f47ca361858 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219bad2b972c45a8930d6f47ca361858 already in progress: creating tablet
I20240507 08:12:19.163434 7079 rpcz_store.cc:276] 0507 08:12:16.902748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.907017 (+ 4269us) service_pool.cc:225] Handling call
0507 08:12:19.161833 (+2254816us) ts_tablet_manager.cc:1120] Deleting tablet 3954a4e0d3104161aad427eb2781d55d
0507 08:12:19.161846 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.162681 (+ 835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2253942}
I20240507 08:12:19.164443 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 399d3d3a9cc84582af16a4f5a695a4ad (DEFAULT_TABLE table=default.Table_ff84c856a98f49c1987c5064a57e91bd [id=30c130e9314b44239e345193c6c7f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.164860 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d3d3a9cc84582af16a4f5a695a4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.165555 7079 tablet_replica.cc:330] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:19.167390 7079 raft_consensus.cc:2229] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:19.169167 5654 consensus_peers.cc:485] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.169678 7079 raft_consensus.cc:2258] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.170934 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 399d3d3a9cc84582af16a4f5a695a4ad (DEFAULT_TABLE table=default.Table_ff84c856a98f49c1987c5064a57e91bd [id=30c130e9314b44239e345193c6c7f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.171314 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d3d3a9cc84582af16a4f5a695a4ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:19.171312 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3954a4e0d3104161aad427eb2781d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3954a4e0d3104161aad427eb2781d55d already in progress: creating tablet
I20240507 08:12:19.171926 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 (DEFAULT_TABLE table=default.Table_f35aa593fc4c45319217c4ce761d30f9 [id=fee54bfb628443198b8f6bffb404658c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.172286 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae07896c98b4d50b3cfae208b46ea76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.177258 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet aae2f13901a146b7a2019709b28930e9
I20240507 08:12:19.177953 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c1935dd9e2a34c49bb3e0021f46bc765
I20240507 08:12:19.180420 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e1bcbadcf2d64f1c99318cda8cddc6e0" } new_table_name: "default.Table_4223bcc97b28449e96a43c1bf917c4ba" modify_external_catalogs: true
I20240507 08:12:19.181636 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6bd4fbf9fea6459893b5a20cf44ff554" } new_table_name: "default.Table_7d29892fe7de4a6b90d5de438680e53c" modify_external_catalogs: true
I20240507 08:12:19.182713 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d03405787efc42a89e644c26fe8f973e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.183172 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d03405787efc42a89e644c26fe8f973e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.184906 7079 ts_tablet_manager.cc:1875] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.187542 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 449d6c8270184ce2918cee8ce731f788 (table default.Table_2c4d993ae928415181f53571b6bf0815 [id=3764a23f5a7847c8b46385bdfaf51f20]). Expected version 1 got 0
I20240507 08:12:19.187773 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet accf88112b0b408e8d24828fc93055ef (table default.Table_0bfaacfdf3354499882aa8bdcc370cac [id=cf35bf92a1e04866a744a6a678948844]). Expected version 1 got 0
I20240507 08:12:19.187978 5259 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 8e6454c4a59d4c4cae1681bcae0da2d7 (table default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3]). Expected version 2 got 0
I20240507 08:12:19.189674 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet aae2f13901a146b7a2019709b28930e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.190559 5683 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 (DEFAULT_TABLE table=default.Table_f35aa593fc4c45319217c4ce761d30f9 [id=fee54bfb628443198b8f6bffb404658c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.190935 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae07896c98b4d50b3cfae208b46ea76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.191596 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 1754588198d54077984f04c9b8f5db33 (DEFAULT_TABLE table=default.Table_d2ac4a0f8696476aab2548d83435ddc6 [id=8e92f79852274c20b2e8e17bf2e5e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.191905 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1754588198d54077984f04c9b8f5db33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.192015 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.192891 7602 raft_consensus.cc:482] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.193025 7602 raft_consensus.cc:504] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.193673 7602 leader_election.cc:288] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.194036 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0e29c9c70e364d598433f81aa779ae4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.194535 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e29c9c70e364d598433f81aa779ae4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.199126 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet aae2f13901a146b7a2019709b28930e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.150344 5843 tablet_bootstrap.cc:492] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.199354 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.199553 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 1754588198d54077984f04c9b8f5db33 (DEFAULT_TABLE table=default.Table_d2ac4a0f8696476aab2548d83435ddc6 [id=8e92f79852274c20b2e8e17bf2e5e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.199561 5548 tablet_service.cc:1467] Processing CreateTablet for tablet eb24b9da423d493d94a182d03ae25dcc (DEFAULT_TABLE table=default.Table_f13cbf54f8414b51bc0105896b80f077 [id=f5753368c0734e35a4163a222b7ad3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.200070 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1754588198d54077984f04c9b8f5db33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.200590 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb24b9da423d493d94a182d03ae25dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.203653 7499 raft_consensus.cc:482] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.203814 7499 raft_consensus.cc:504] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.204464 7499 leader_election.cc:288] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.206048 5843 tablet_bootstrap.cc:654] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.210175 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.210305 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 34865439bf8242da8081077210f02c35 (DEFAULT_TABLE table=default.Table_817b15f93fff45d69d52c60a2300e671 [id=7f7a565e4077427b87f26fc2cefe455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.210666 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34865439bf8242da8081077210f02c35. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:19.210819 5378 leader_election.cc:341] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.212917 5841 ts_tablet_manager.cc:1875] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.215620 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d3f120d9fbf542d39508903883ecb5e6
I20240507 08:12:19.216022 5256 catalog_manager.cc:6554] ReplaceTablet: tablet d3f120d9fbf542d39508903883ecb5e6 deleted and replaced by tablet 950b7ea014b14037b0ed6babb5963db5
I20240507 08:12:19.216926 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f120d9fbf542d39508903883ecb5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 950b7ea014b14037b0ed6babb5963db5) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.220955 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1b17db44e5fc453daefcada73f3e7f24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.221508 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b17db44e5fc453daefcada73f3e7f24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.222010 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6804b703b4a240d389ddf5b42892ed4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.222494 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6804b703b4a240d389ddf5b42892ed4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.222584 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c4a924e93759482bba1dfca1998c326c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.222884 7673 tablet.cc:1722] T 0c4cd7b56d4e468eb754d9653117d023 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20240507 08:12:19.223021 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4a924e93759482bba1dfca1998c326c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:19.227226 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 33a58839996b414d805c6b2873254562 consensus state has no leader
I20240507 08:12:19.227600 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_56f7efcd33244fe0b455eabef3d06f68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.228050 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56f7efcd33244fe0b455eabef3d06f68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:19.230005 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ececf80512074304ab59db6cbb83f00b consensus state has no leader
I20240507 08:12:19.230345 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bac149077ee54ee3ba57739c3b704a42" } new_table_name: "default.Table_82e54470d4fa4afc8451693317fc1d4e" modify_external_catalogs: true
I20240507 08:12:19.233009 5560 tablet_service.cc:1467] Processing CreateTablet for tablet bae3b5e5c5f6489a922ba32a68755fa8 (DEFAULT_TABLE table=default.Table_16a1c2b0c7bd4504a1d2c6b472ccecbf [id=114fd466b5bf4b1b882d2202d5a27702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.233094 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe1592caef4211ac6497c57cf79624 (DEFAULT_TABLE table=default.Table_1bc794eff3b343839470ca0506214d11 [id=57c1e51e0367469d8e8d794625ef53b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.233527 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efe1592caef4211ac6497c57cf79624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.236479 7649 consensus_queue.cc:1035] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.238274 5840 tablet_bootstrap.cc:654] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.240536 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae3b5e5c5f6489a922ba32a68755fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.240586 5698 tablet_service.cc:1467] Processing CreateTablet for tablet eb24b9da423d493d94a182d03ae25dcc (DEFAULT_TABLE table=default.Table_f13cbf54f8414b51bc0105896b80f077 [id=f5753368c0734e35a4163a222b7ad3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.240988 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb24b9da423d493d94a182d03ae25dcc. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:19.241547 5517 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:12:19.246814 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:19.247951 5699 tablet_service.cc:1467] Processing CreateTablet for tablet cd29341c0404492d97eeade187f757df (DEFAULT_TABLE table=default.Table_5f53576c761847f081c511b4d5c067ef [id=f05d7160c6b74aa3baa1319ed3775551]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.248366 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd29341c0404492d97eeade187f757df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.249126 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 34865439bf8242da8081077210f02c35 (DEFAULT_TABLE table=default.Table_817b15f93fff45d69d52c60a2300e671 [id=7f7a565e4077427b87f26fc2cefe455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.248484 7079 ts_tablet_manager.cc:1888] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.249457 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f120d9fbf542d39508903883ecb5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 950b7ea014b14037b0ed6babb5963db5) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.251215 7079 log.cc:1196] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/3a02a5a493bc47cdbb6f7bd5b5c28e7f
I20240507 08:12:19.255795 7079 ts_tablet_manager.cc:1909] T 3a02a5a493bc47cdbb6f7bd5b5c28e7f P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:19.261855 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 0b84404f78334ff49fff01b4b95e651a (DEFAULT_TABLE table=default.Table_165fc764f58349838a6022d9320583ac [id=b23f726cfd094c04ad6acfa9ea424f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.262257 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b84404f78334ff49fff01b4b95e651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.263211 5839 tablet_bootstrap.cc:492] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.263545 5839 ts_tablet_manager.cc:1367] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.168s user 0.012s sys 0.020s
I20240507 08:12:19.265527 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34865439bf8242da8081077210f02c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.266253 5839 raft_consensus.cc:348] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.266506 5693 tablet_service.cc:1467] Processing CreateTablet for tablet bae3b5e5c5f6489a922ba32a68755fa8 (DEFAULT_TABLE table=default.Table_16a1c2b0c7bd4504a1d2c6b472ccecbf [id=114fd466b5bf4b1b882d2202d5a27702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.266719 5839 raft_consensus.cc:374] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.266865 5839 raft_consensus.cc:729] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.266940 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae3b5e5c5f6489a922ba32a68755fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.267428 5839 consensus_queue.cc:260] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.269821 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe1592caef4211ac6497c57cf79624 (DEFAULT_TABLE table=default.Table_1bc794eff3b343839470ca0506214d11 [id=57c1e51e0367469d8e8d794625ef53b0]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:19.271253 5377 leader_election.cc:341] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.271384 5377 leader_election.cc:302] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.271608 7499 raft_consensus.cc:2726] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:19.273260 5654 consensus_peers.cc:485] T fed715ba993a4f0c9e54b209ef031981 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.280453 5839 ts_tablet_manager.cc:1398] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240507 08:12:19.285060 5552 tablet_service.cc:1467] Processing CreateTablet for tablet cd29341c0404492d97eeade187f757df (DEFAULT_TABLE table=default.Table_5f53576c761847f081c511b4d5c067ef [id=f05d7160c6b74aa3baa1319ed3775551]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.285538 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd29341c0404492d97eeade187f757df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.293654 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1198) took 2465 ms. Trace:
I20240507 08:12:19.293758 7079 rpcz_store.cc:276] 0507 08:12:16.827815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.926392 (+ 98577us) service_pool.cc:225] Handling call
0507 08:12:19.165060 (+2238668us) ts_tablet_manager.cc:1120] Deleting tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f
0507 08:12:19.165291 (+ 231us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.214904 (+ 49613us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.245485 (+ 30581us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.293632 (+ 48147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2175339}
I20240507 08:12:19.294209 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1207) took 2232 ms. Trace:
I20240507 08:12:19.294292 7079 rpcz_store.cc:276] 0507 08:12:17.061962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.062011 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:19.294046 (+2232035us) ts_tablet_manager.cc:1120] Deleting tablet c1b0a2dddb15451a8e6230ffb52d5302
0507 08:12:19.294057 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.294192 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2231851}
I20240507 08:12:19.294565 7079 tablet_replica.cc:330] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:19.294400 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 3a02a5a493bc47cdbb6f7bd5b5c28e7f (table default.Table_329b8968956c49bdb8e75c6fb49ff3b1 [id=bb41979b47e74d1ca9ecb58cf2c74b42]) successfully deleted
I20240507 08:12:19.294750 7079 raft_consensus.cc:2229] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.294929 7079 raft_consensus.cc:2258] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.295500 7079 ts_tablet_manager.cc:1875] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.295609 5841 ts_tablet_manager.cc:1888] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.295725 5841 log.cc:1196] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b8c6f552832a45c4a2e28f40797db8fa
I20240507 08:12:19.296089 5841 ts_tablet_manager.cc:1909] T b8c6f552832a45c4a2e28f40797db8fa P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:19.296388 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c1b0a2dddb15451a8e6230ffb52d5302 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b0a2dddb15451a8e6230ffb52d5302 already in progress: creating tablet
I20240507 08:12:19.297497 5839 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.301286 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 0b84404f78334ff49fff01b4b95e651a (DEFAULT_TABLE table=default.Table_165fc764f58349838a6022d9320583ac [id=b23f726cfd094c04ad6acfa9ea424f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.302342 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b84404f78334ff49fff01b4b95e651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.306353 5843 tablet_bootstrap.cc:492] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.307958 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet a905e345412541558b30c629a2eb86c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.307592 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 01820ecad55a4097bc5917a8f7949fe7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.308125 5843 ts_tablet_manager.cc:1367] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.159s user 0.006s sys 0.000s
I20240507 08:12:19.309028 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efe1592caef4211ac6497c57cf79624. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.311985 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_904d0fe58daf4a51a076b0b29aa4c942"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.312626 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_904d0fe58daf4a51a076b0b29aa4c942 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.312836 7499 raft_consensus.cc:482] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.312983 7499 raft_consensus.cc:504] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.314319 7499 leader_election.cc:288] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.314929 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 399d3d3a9cc84582af16a4f5a695a4ad (DEFAULT_TABLE table=default.Table_ff84c856a98f49c1987c5064a57e91bd [id=30c130e9314b44239e345193c6c7f057]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.315302 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399d3d3a9cc84582af16a4f5a695a4ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.318579 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d608eb93e54b43d2b21ba5c735d0f778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.319054 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d608eb93e54b43d2b21ba5c735d0f778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.319550 5422 tablet_service.cc:1467] Processing CreateTablet for tablet eb24b9da423d493d94a182d03ae25dcc (DEFAULT_TABLE table=default.Table_f13cbf54f8414b51bc0105896b80f077 [id=f5753368c0734e35a4163a222b7ad3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.319890 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb24b9da423d493d94a182d03ae25dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.320497 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1a1a5e38eff2435c9be461ea3ed65ca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.320850 5843 raft_consensus.cc:348] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.321036 5843 raft_consensus.cc:374] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.321090 5843 raft_consensus.cc:729] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.321130 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet c1935dd9e2a34c49bb3e0021f46bc765 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.321269 5843 consensus_queue.cc:260] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.321516 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 1754588198d54077984f04c9b8f5db33 (DEFAULT_TABLE table=default.Table_d2ac4a0f8696476aab2548d83435ddc6 [id=8e92f79852274c20b2e8e17bf2e5e45a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.321851 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1754588198d54077984f04c9b8f5db33. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.322077 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 34865439bf8242da8081077210f02c35 (DEFAULT_TABLE table=default.Table_817b15f93fff45d69d52c60a2300e671 [id=7f7a565e4077427b87f26fc2cefe455b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.322340 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1210) took 3040 ms. Trace:
I20240507 08:12:19.321812 5843 ts_tablet_manager.cc:1398] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20240507 08:12:19.322506 5418 tablet_service.cc:1514] Processing DeleteTablet for tablet d3f120d9fbf542d39508903883ecb5e6 with delete_type TABLET_DATA_DELETED (replaced by tablet 950b7ea014b14037b0ed6babb5963db5) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.322432 5841 rpcz_store.cc:276] 0507 08:12:16.281496 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.281560 (+ 64us) service_pool.cc:225] Handling call
0507 08:12:19.144740 (+2863180us) ts_tablet_manager.cc:1120] Deleting tablet b8c6f552832a45c4a2e28f40797db8fa
0507 08:12:19.144752 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.215778 (+ 71026us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.295496 (+ 79718us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.322312 (+ 26816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25560,"tablet-delete.queue_time_us":2862981}
I20240507 08:12:19.322780 5420 tablet_service.cc:1467] Processing CreateTablet for tablet bae3b5e5c5f6489a922ba32a68755fa8 (DEFAULT_TABLE table=default.Table_16a1c2b0c7bd4504a1d2c6b472ccecbf [id=114fd466b5bf4b1b882d2202d5a27702]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.322917 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b8c6f552832a45c4a2e28f40797db8fa (table default.Table_6500147915384d518c459e558a9ce1d6 [id=98ca5c705ae24203aa9e3946bcbc3bbf]) successfully deleted
I20240507 08:12:19.323089 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 5efe1592caef4211ac6497c57cf79624 (DEFAULT_TABLE table=default.Table_1bc794eff3b343839470ca0506214d11 [id=57c1e51e0367469d8e8d794625ef53b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.322434 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 0b84404f78334ff49fff01b4b95e651a (DEFAULT_TABLE table=default.Table_165fc764f58349838a6022d9320583ac [id=b23f726cfd094c04ad6acfa9ea424f84]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.321815 5409 tablet_service.cc:1467] Processing CreateTablet for tablet cd29341c0404492d97eeade187f757df (DEFAULT_TABLE table=default.Table_5f53576c761847f081c511b4d5c067ef [id=f05d7160c6b74aa3baa1319ed3775551]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.323619 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 34865439bf8242da8081077210f02c35. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.326020 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_8e2edbe2bb52425bbde255742f34f784" } new_table_name: "default.Table_7bbfcef77950434d9b09839fb0bac453" modify_external_catalogs: true
I20240507 08:12:19.327335 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 (DEFAULT_TABLE table=default.Table_f35aa593fc4c45319217c4ce761d30f9 [id=fee54bfb628443198b8f6bffb404658c]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:19.327728 5520 leader_election.cc:341] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:12:19.327852 5520 leader_election.cc:302] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:19.328214 7602 raft_consensus.cc:2726] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.328439 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet aae2f13901a146b7a2019709b28930e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.329025 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_76d0ef5741d34f578a0810ddcde0bc44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.329870 5841 tablet_replica.cc:330] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:19.329936 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ae07896c98b4d50b3cfae208b46ea76. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.330051 5841 raft_consensus.cc:2229] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.330196 5841 raft_consensus.cc:2258] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.330850 5841 ts_tablet_manager.cc:1875] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.333351 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5efe1592caef4211ac6497c57cf79624. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:19.333703 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a1a5e38eff2435c9be461ea3ed65ca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.335914 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.336738 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5e6f4cb669d74f3594bba624c848009d" } modify_external_catalogs: true
I20240507 08:12:19.336891 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5e6f4cb669d74f3594bba624c848009d" } modify_external_catalogs: true
I20240507 08:12:19.337226 5841 ts_tablet_manager.cc:1888] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.337329 5841 log.cc:1196] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8bb62a72ed2248f6886823b9c64ff4c6
I20240507 08:12:19.337729 5841 ts_tablet_manager.cc:1909] T 8bb62a72ed2248f6886823b9c64ff4c6 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:19.338163 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b84404f78334ff49fff01b4b95e651a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.338444 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1211) took 3022 ms. Trace:
W20240507 08:12:19.338629 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d0ef5741d34f578a0810ddcde0bc44 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.338534 5841 rpcz_store.cc:276] 0507 08:12:16.316382 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.316492 (+ 110us) service_pool.cc:225] Handling call
0507 08:12:19.322817 (+3006325us) ts_tablet_manager.cc:1120] Deleting tablet 8bb62a72ed2248f6886823b9c64ff4c6
0507 08:12:19.329818 (+ 7001us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.334369 (+ 4551us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.337154 (+ 2785us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.338422 (+ 1268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3006006}
I20240507 08:12:19.338946 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1208) took 3162 ms. Trace:
I20240507 08:12:19.338948 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 8bb62a72ed2248f6886823b9c64ff4c6 (table default.Table_f9c3b5637d424b1199065768db5c961a [id=866842c12bfe4b5aa14e08e8e3e226e3]) successfully deleted
I20240507 08:12:19.339020 5841 rpcz_store.cc:276] 0507 08:12:16.175982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.344418 (+168436us) service_pool.cc:225] Handling call
0507 08:12:19.338814 (+2994396us) ts_tablet_manager.cc:1120] Deleting tablet b15c8ad2a8324c75a1ef49e219b962cf
0507 08:12:19.338824 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.338928 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2993762}
I20240507 08:12:19.339277 5841 tablet_replica.cc:330] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:19.339327 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b15c8ad2a8324c75a1ef49e219b962cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b15c8ad2a8324c75a1ef49e219b962cf already in progress: creating tablet
I20240507 08:12:19.339465 5841 raft_consensus.cc:2229] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.339654 5841 raft_consensus.cc:2258] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:19.340005 5378 leader_election.cc:341] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.341948 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bae3b5e5c5f6489a922ba32a68755fa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.342141 6513 ts_tablet_manager.cc:1888] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.342239 6513 log.cc:1196] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:19.342581 6513 ts_tablet_manager.cc:1909] T 767905763ba14cb8bbc52151ae9f17e6 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.342886 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1178) took 3828 ms. Trace:
I20240507 08:12:19.342963 6513 rpcz_store.cc:276] 0507 08:12:15.514481 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.515020 (+ 539us) service_pool.cc:225] Handling call
0507 08:12:19.092269 (+3577249us) ts_tablet_manager.cc:1120] Deleting tablet 767905763ba14cb8bbc52151ae9f17e6
0507 08:12:19.092281 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.252186 (+159905us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.342075 (+ 89889us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.342869 (+ 794us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3545997}
I20240507 08:12:19.343322 6513 tablet_replica.cc:330] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.343508 6513 raft_consensus.cc:2229] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:19.343834 6513 raft_consensus.cc:2258] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.344529 5261 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_56975594d01a4159aeafef5258ff445d [id=f274fff64ffc4f0eac28cbb1115fc0a3] alter complete (version 2)
I20240507 08:12:19.345661 6513 ts_tablet_manager.cc:1875] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.346045 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:19.346503 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd29341c0404492d97eeade187f757df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.347105 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 767905763ba14cb8bbc52151ae9f17e6 (table default.Table_4d38bf47908640beb3f6d94ad4446717 [id=84ee77d848974c2d8ccd5d9d518e1ec2]) successfully deleted
I20240507 08:12:19.349552 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_db5191b1f46846118fc8702ca6d48fe8" } new_table_name: "default.Table_697e51662b0b4accadb633f6980acf76" modify_external_catalogs: true
I20240507 08:12:19.353016 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c6f552832a45c4a2e28f40797db8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.353873 5261 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a5bc6a7273624dceb2e737e0f7df622c" } modify_external_catalogs: true
I20240507 08:12:19.354044 5261 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a5bc6a7273624dceb2e737e0f7df622c" } modify_external_catalogs: true
I20240507 08:12:19.358004 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.358949 5843 tablet_bootstrap.cc:492] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.360347 5843 tablet_bootstrap.cc:654] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:19.360926 5377 leader_election.cc:341] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:19.361001 5654 consensus_peers.cc:485] T 0c4cd7b56d4e468eb754d9653117d023 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:19.361042 5377 leader_election.cc:302] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.361404 5683 tablet_service.cc:1514] Processing DeleteTablet for tablet e7eef600626a4961ab1fae27162537bb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.361567 7499 raft_consensus.cc:2726] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.361636 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet a5976eca006448e0b9f2573acd738bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 5481ddbcd31449c4b44afb87a97138bd) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.362582 5698 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b13fbddcab5460db69f43e63f490fba with delete_type TABLET_DATA_DELETED (replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.364049 5841 ts_tablet_manager.cc:1875] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.366127 5840 tablet_bootstrap.cc:492] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.366274 5840 ts_tablet_manager.cc:1367] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.205s user 0.018s sys 0.000s
I20240507 08:12:19.367493 7079 ts_tablet_manager.cc:1888] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:19.367358 5840 raft_consensus.cc:348] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.367614 7079 log.cc:1196] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/e17502bb4ac64f6ca9c77a3b00ab9f4a
I20240507 08:12:19.367609 5840 raft_consensus.cc:374] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.367660 5840 raft_consensus.cc:729] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.367986 7079 ts_tablet_manager.cc:1909] T e17502bb4ac64f6ca9c77a3b00ab9f4a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:19.367872 5840 consensus_queue.cc:260] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.371891 5840 ts_tablet_manager.cc:1398] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240507 08:12:19.372140 5840 tablet_bootstrap.cc:492] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.375351 5841 ts_tablet_manager.cc:1888] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20240507 08:12:19.376538 5656 consensus_peers.cc:485] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99e4514039304f0696c69a057db0ea6e. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.383664 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1208) took 2305 ms. Trace:
I20240507 08:12:19.383764 7079 rpcz_store.cc:276] 0507 08:12:17.078542 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.078593 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:19.294521 (+2215928us) ts_tablet_manager.cc:1120] Deleting tablet e17502bb4ac64f6ca9c77a3b00ab9f4a
0507 08:12:19.294532 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.318029 (+ 23497us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.367380 (+ 49351us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.383641 (+ 16261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2215746}
I20240507 08:12:19.384392 5841 log.cc:1196] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/43644b4405f44bfd9a1afcaedbd68db5
I20240507 08:12:19.388160 5840 tablet_bootstrap.cc:654] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.390393 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]) successfully deleted
I20240507 08:12:19.391505 5843 tablet_bootstrap.cc:492] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.391635 5843 ts_tablet_manager.cc:1367] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20240507 08:12:19.392289 5843 raft_consensus.cc:348] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.392482 5843 raft_consensus.cc:374] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.392539 5843 raft_consensus.cc:729] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.392724 5843 consensus_queue.cc:260] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.393314 5843 ts_tablet_manager.cc:1398] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:19.393671 5843 tablet_bootstrap.cc:492] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.394898 5843 tablet_bootstrap.cc:654] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.401499 6513 ts_tablet_manager.cc:1888] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.402390 5843 tablet_bootstrap.cc:492] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.402561 5843 ts_tablet_manager.cc:1367] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:19.403204 5843 raft_consensus.cc:348] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.403448 5843 raft_consensus.cc:374] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.403548 5843 raft_consensus.cc:729] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.403807 5843 consensus_queue.cc:260] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.404361 5843 ts_tablet_manager.cc:1398] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.404650 5843 tablet_bootstrap.cc:492] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.405992 5843 tablet_bootstrap.cc:654] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.412137 7201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap replaying log segment 1/1 (101.6K/158.6K this segment, stats: ops{read=720 overwritten=0 applied=244 ignored=0} inserts{seen=226 ignored=0} mutations{seen=262 ignored=0} orphaned_commits=0)
W20240507 08:12:19.413815 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e54273ed92f640d98b853a7661ae3e26 consensus state has no leader
I20240507 08:12:19.414098 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65d962c3598a4be5be40be25fc685388" } modify_external_catalogs: true
I20240507 08:12:19.414237 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_65d962c3598a4be5be40be25fc685388" } modify_external_catalogs: true
I20240507 08:12:19.415603 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9bdc371ba30a4c879ec337f2a1d3e90e" } modify_external_catalogs: true
I20240507 08:12:19.415738 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9bdc371ba30a4c879ec337f2a1d3e90e" } modify_external_catalogs: true
W20240507 08:12:19.418804 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6d972e15fa840668e74ac1416c2e2d9 consensus state has no leader
I20240507 08:12:19.419085 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5e4b55cf0914d33a568f30399e6a052" } modify_external_catalogs: true
I20240507 08:12:19.419215 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_b5e4b55cf0914d33a568f30399e6a052" } modify_external_catalogs: true
I20240507 08:12:19.418313 7649 raft_consensus.cc:482] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.420997 6675 raft_consensus.cc:482] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.421917 6513 log.cc:1196] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/53bc9bf1c04e4015a152cb750184c9ce
I20240507 08:12:19.422310 6513 ts_tablet_manager.cc:1909] T 53bc9bf1c04e4015a152cb750184c9ce P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.423370 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet ece0e5eae0d949149dd3bd3f4bbd4ca7
I20240507 08:12:19.424129 7602 raft_consensus.cc:482] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.424259 7602 raft_consensus.cc:504] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.424870 7602 leader_election.cc:288] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.425096 6675 raft_consensus.cc:504] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.425835 6675 leader_election.cc:288] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.421555 7649 raft_consensus.cc:504] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.426430 7649 leader_election.cc:288] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:19.426502 5843 tablet_bootstrap.cc:492] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.426617 5843 ts_tablet_manager.cc:1367] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20240507 08:12:19.427183 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:19.427230 5843 raft_consensus.cc:348] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.427389 5443 raft_consensus.cc:2454] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:19.427448 5843 raft_consensus.cc:374] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.427506 5843 raft_consensus.cc:729] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.427731 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:19.427703 5843 consensus_queue.cc:260] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.427908 5442 raft_consensus.cc:2454] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:19.428184 5843 ts_tablet_manager.cc:1398] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20240507 08:12:19.428367 5520 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:19.428716 5656 leader_election.cc:302] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:19.428977 5656 leader_election.cc:302] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:19.429167 6675 raft_consensus.cc:2781] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:19.429247 6675 raft_consensus.cc:482] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:19.429307 6675 raft_consensus.cc:3037] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:19.429287 5517 leader_election.cc:341] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:19.429383 5517 leader_election.cc:302] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:19.429795 7649 raft_consensus.cc:2781] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:19.429801 7602 raft_consensus.cc:2726] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.429879 7649 raft_consensus.cc:482] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:19.429942 7649 raft_consensus.cc:3037] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.430805 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.432744 7649 raft_consensus.cc:504] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.433318 7649 leader_election.cc:288] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.435493 6675 raft_consensus.cc:504] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.435597 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.436096 6675 leader_election.cc:288] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:19.436487 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:19.436637 5443 raft_consensus.cc:3037] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.437620 7079 tablet_replica.cc:330] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:19.437678 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:19.437817 5442 raft_consensus.cc:3037] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.438011 7079 raft_consensus.cc:2229] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.438231 7079 raft_consensus.cc:2258] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.439074 7079 ts_tablet_manager.cc:1875] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.440222 5443 raft_consensus.cc:2454] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:19.440629 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1180) took 3925 ms. Trace:
I20240507 08:12:19.440649 7602 raft_consensus.cc:482] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.440757 7602 raft_consensus.cc:504] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.440716 6513 rpcz_store.cc:276] 0507 08:12:15.514659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.516898 (+ 2239us) service_pool.cc:225] Handling call
0507 08:12:19.343279 (+3826381us) ts_tablet_manager.cc:1120] Deleting tablet 53bc9bf1c04e4015a152cb750184c9ce
0507 08:12:19.343289 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.381822 (+ 38533us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.401403 (+ 19581us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.440611 (+ 39208us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3796970}
I20240507 08:12:19.441059 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1179) took 3926 ms. Trace:
I20240507 08:12:19.441128 6513 rpcz_store.cc:276] 0507 08:12:15.514577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.514787 (+ 210us) service_pool.cc:225] Handling call
0507 08:12:19.440972 (+3926185us) ts_tablet_manager.cc:1120] Deleting tablet 1c7c39ec9fec4331afabb39d1483ef57
0507 08:12:19.440980 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.441042 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3894215}
I20240507 08:12:19.441298 7602 leader_election.cc:288] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.441390 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1184) took 3906 ms. Trace:
I20240507 08:12:19.441462 6513 rpcz_store.cc:276] 0507 08:12:15.534536 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.549503 (+ 14967us) service_pool.cc:225] Handling call
0507 08:12:19.441317 (+3891814us) ts_tablet_manager.cc:1120] Deleting tablet 35e34b60b78b402faf51e003c97db787
0507 08:12:19.441324 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.441376 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3891569}
I20240507 08:12:19.441742 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1187) took 3891 ms. Trace:
I20240507 08:12:19.444181 5442 raft_consensus.cc:2454] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:19.444873 5656 leader_election.cc:302] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:19.445115 6675 raft_consensus.cc:2781] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:19.445158 5656 leader_election.cc:302] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:19.445230 6675 raft_consensus.cc:686] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:19.445454 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:19.445488 6675 consensus_queue.cc:237] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.445879 7358 raft_consensus.cc:2781] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:19.446007 7358 raft_consensus.cc:686] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:19.441813 6513 rpcz_store.cc:276] 0507 08:12:15.550266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.550309 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:19.441668 (+3891359us) ts_tablet_manager.cc:1120] Deleting tablet 1037c3c0af5e464bb23a311e552fcf21
0507 08:12:19.441677 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.441728 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3891216}
I20240507 08:12:19.446233 7358 consensus_queue.cc:237] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.446532 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1186) took 3896 ms. Trace:
I20240507 08:12:19.446625 6513 rpcz_store.cc:276] 0507 08:12:15.550174 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.551218 (+ 1044us) service_pool.cc:225] Handling call
0507 08:12:19.446435 (+3895217us) ts_tablet_manager.cc:1120] Deleting tablet 44ec0526e1914221bc22b3e75ea01e0d
0507 08:12:19.446447 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.446515 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3895052}
I20240507 08:12:19.447136 6513 tablet_replica.cc:330] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.447371 6513 raft_consensus.cc:2229] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.448931 6513 raft_consensus.cc:2258] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.450210 6513 ts_tablet_manager.cc:1875] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.454109 5841 ts_tablet_manager.cc:1909] T 43644b4405f44bfd9a1afcaedbd68db5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:19.454766 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 53bc9bf1c04e4015a152cb750184c9ce (table default.Table_5b668756389c48ef943681187557167a [id=5331470f800a425eb58c185aef9bd183]) successfully deleted
W20240507 08:12:19.455011 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1c7c39ec9fec4331afabb39d1483ef57 because the tablet was not found. No further retry: Not found: Tablet not found: 1c7c39ec9fec4331afabb39d1483ef57
W20240507 08:12:19.455157 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 35e34b60b78b402faf51e003c97db787 because the tablet was not found. No further retry: Not found: Tablet not found: 35e34b60b78b402faf51e003c97db787
W20240507 08:12:19.455550 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 1037c3c0af5e464bb23a311e552fcf21 because the tablet was not found. No further retry: Not found: Tablet not found: 1037c3c0af5e464bb23a311e552fcf21
W20240507 08:12:19.455749 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 44ec0526e1914221bc22b3e75ea01e0d because the tablet was not found. No further retry: Not found: Tablet not found: 44ec0526e1914221bc22b3e75ea01e0d
I20240507 08:12:19.456804 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.457334 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.460100 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3fce98f472ef46589dbea6b6aefa6572" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:19.461660 5581 raft_consensus.cc:3037] T 2754fc46c36346f9828198e3bb6ed3a7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:19.465528 5654 leader_election.cc:341] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.465690 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7eef600626a4961ab1fae27162537bb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:19.465889 5654 leader_election.cc:341] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:19.466229 5654 leader_election.cc:341] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.466754 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_faf3a930a35b4a80aa6be69297d5ed7b" } new_table_name: "default.Table_a25dcd1359ea4149b0762803cba8b949" modify_external_catalogs: true
I20240507 08:12:19.471519 6675 consensus_queue.cc:1035] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.472285 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ea12b1a814f546a19315986dde83e3db" } new_table_name: "default.Table_6c303ed1409e4f65b3338f9d5c02d424" modify_external_catalogs: true
I20240507 08:12:19.454658 5843 tablet_bootstrap.cc:492] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.475768 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1212) took 2986 ms. Trace:
W20240507 08:12:19.476095 5656 consensus_peers.cc:485] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4e5439844f384513b4719cfe4f4e5d9b. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:19.478811 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1314775fa5df438ab3aa1eb0075e7f78 has no consensus state
W20240507 08:12:19.479920 5654 leader_election.cc:341] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.470084 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba28c36f91e141f4a1372e7544b44281" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:19.476790 5843 tablet_bootstrap.cc:654] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.481700 5520 leader_election.cc:302] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:19.482111 7602 raft_consensus.cc:2726] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.482959 5839 tablet_bootstrap.cc:654] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.484498 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.481469 5841 rpcz_store.cc:276] 0507 08:12:16.488950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.489007 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:19.339239 (+2850232us) ts_tablet_manager.cc:1120] Deleting tablet 43644b4405f44bfd9a1afcaedbd68db5
0507 08:12:19.339247 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.367070 (+ 27823us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.375245 (+ 8175us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.475747 (+100502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2850064}
I20240507 08:12:19.486353 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1213) took 2947 ms. Trace:
I20240507 08:12:19.486420 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]) successfully deleted
I20240507 08:12:19.487182 5841 rpcz_store.cc:276] 0507 08:12:16.538780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.538838 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:19.486255 (+2947417us) ts_tablet_manager.cc:1120] Deleting tablet 85dc58133d2b4f5496b4e570ae6494c0
0507 08:12:19.486264 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.486338 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2947224}
I20240507 08:12:19.487541 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1214) took 2947 ms. Trace:
I20240507 08:12:19.487614 5841 rpcz_store.cc:276] 0507 08:12:16.539637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.541195 (+ 1558us) service_pool.cc:225] Handling call
0507 08:12:19.487466 (+2946271us) ts_tablet_manager.cc:1120] Deleting tablet 767905763ba14cb8bbc52151ae9f17e6
0507 08:12:19.487475 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.487528 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2946123}
W20240507 08:12:19.487680 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 85dc58133d2b4f5496b4e570ae6494c0 because the tablet was not found. No further retry: Not found: Tablet not found: 85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:19.487934 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1217) took 2941 ms. Trace:
I20240507 08:12:19.488003 5841 rpcz_store.cc:276] 0507 08:12:16.546050 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.546102 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:19.487816 (+2941714us) ts_tablet_manager.cc:1120] Deleting tablet 310c601abead42e0bd2a2df1a0e16ec2
0507 08:12:19.487826 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.487919 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2941558}
W20240507 08:12:19.488130 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 767905763ba14cb8bbc52151ae9f17e6 because the tablet was not found. No further retry: Not found: Tablet not found: 767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:19.488243 5841 tablet_replica.cc:330] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:19.488422 5841 raft_consensus.cc:2229] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:19.488575 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 310c601abead42e0bd2a2df1a0e16ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310c601abead42e0bd2a2df1a0e16ec2 already in progress: creating tablet
I20240507 08:12:19.488730 5841 raft_consensus.cc:2258] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.489418 5841 ts_tablet_manager.cc:1875] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.494293 5840 tablet_bootstrap.cc:492] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.494449 5840 ts_tablet_manager.cc:1367] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.122s user 0.005s sys 0.004s
I20240507 08:12:19.498023 5843 tablet_bootstrap.cc:492] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.498263 5843 ts_tablet_manager.cc:1367] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.006s
I20240507 08:12:19.498534 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet e797fee2c43c4461a5cb9e2f0a2bb0fc
I20240507 08:12:19.499279 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.499509 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.499146 5843 raft_consensus.cc:348] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.499820 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.499969 5843 raft_consensus.cc:374] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.500104 5843 raft_consensus.cc:729] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.500475 5261 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b82da7a22025499b928406579ae43a50
I20240507 08:12:19.500464 5843 consensus_queue.cc:260] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.501318 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet b82da7a22025499b928406579ae43a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.501833 5843 ts_tablet_manager.cc:1398] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240507 08:12:19.501819 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet b82da7a22025499b928406579ae43a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.502107 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet b82da7a22025499b928406579ae43a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.502238 5843 tablet_bootstrap.cc:492] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.502431 5840 raft_consensus.cc:348] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.502609 5840 raft_consensus.cc:374] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.502662 5840 raft_consensus.cc:729] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.503895 5843 tablet_bootstrap.cc:654] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.508020 5840 consensus_queue.cc:260] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.510362 6513 ts_tablet_manager.cc:1888] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.510480 6513 log.cc:1196] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/f2de26190bc44916809856d84ad68345
I20240507 08:12:19.510771 6513 ts_tablet_manager.cc:1909] T f2de26190bc44916809856d84ad68345 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.511091 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1185) took 3961 ms. Trace:
I20240507 08:12:19.511179 6513 rpcz_store.cc:276] 0507 08:12:15.549885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.549928 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:19.446910 (+3896982us) ts_tablet_manager.cc:1120] Deleting tablet f2de26190bc44916809856d84ad68345
0507 08:12:19.447092 (+ 182us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.478700 (+ 31608us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.510263 (+ 31563us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.511074 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3888247}
I20240507 08:12:19.511547 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1189) took 3842 ms. Trace:
I20240507 08:12:19.513280 6675 raft_consensus.cc:482] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.513305 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet f2de26190bc44916809856d84ad68345 (table default.Table_f2eabb2165474aadb0528bf86cbc1718 [id=ff9529d9a26e41cb9ebf472b849d6764]) successfully deleted
I20240507 08:12:19.513427 6675 raft_consensus.cc:504] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.514020 6675 leader_election.cc:288] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:19.514083 7650 tablet.cc:1722] T 6f3bdb777b974524b5673a3f02883107 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:19.514871 5443 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:19.514976 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:19.515129 5443 raft_consensus.cc:2454] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:19.515499 5654 leader_election.cc:341] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.515524 7673 tablet.cc:1722] T 6f3bdb777b974524b5673a3f02883107 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240507 08:12:19.515967 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a769e920141f4ba288ebf85e3a18e791 consensus state has no leader
I20240507 08:12:19.516422 5656 leader_election.cc:302] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:19.516747 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0f5ba0221304453a00bb779aee676d4 consensus state has no leader
I20240507 08:12:19.516996 5843 tablet_bootstrap.cc:492] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.517064 7656 raft_consensus.cc:2781] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:19.517149 7656 raft_consensus.cc:482] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:19.517163 5843 ts_tablet_manager.cc:1367] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20240507 08:12:19.517210 7656 raft_consensus.cc:3037] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.517815 5843 raft_consensus.cc:348] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.517972 5843 raft_consensus.cc:374] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.518026 5843 raft_consensus.cc:729] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.518255 5843 consensus_queue.cc:260] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.519218 5840 ts_tablet_manager.cc:1398] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20240507 08:12:19.519265 5843 ts_tablet_manager.cc:1398] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.519492 5843 tablet_bootstrap.cc:492] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.519493 5840 tablet_bootstrap.cc:492] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.519575 7656 raft_consensus.cc:504] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.520246 7656 leader_election.cc:288] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:19.521010 5840 tablet_bootstrap.cc:654] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.521869 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:19.522351 5843 tablet_bootstrap.cc:654] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:19.528678 5654 leader_election.cc:341] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.534471 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e37b73134cf42ad84b6eec90c9b805f" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:19.511631 6513 rpcz_store.cc:276] 0507 08:12:15.668605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.668659 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:19.511437 (+3842778us) ts_tablet_manager.cc:1120] Deleting tablet 62895377a8ac4506a2f4e10039dbdd74
0507 08:12:19.511446 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.511530 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3842614}
I20240507 08:12:19.534833 5442 raft_consensus.cc:3037] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:19.535130 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 62895377a8ac4506a2f4e10039dbdd74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62895377a8ac4506a2f4e10039dbdd74 already in progress: creating tablet
I20240507 08:12:19.535480 6513 tablet_replica.cc:330] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.535689 6513 raft_consensus.cc:2229] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:19.535841 5843 tablet_bootstrap.cc:492] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.535964 5843 ts_tablet_manager.cc:1367] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20240507 08:12:19.536054 6513 raft_consensus.cc:2258] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.536610 6513 ts_tablet_manager.cc:1875] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.536631 5843 raft_consensus.cc:348] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.536803 5843 raft_consensus.cc:374] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.536863 5843 raft_consensus.cc:729] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.537081 5843 consensus_queue.cc:260] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.537693 5843 ts_tablet_manager.cc:1398] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:19.538475 5843 tablet_bootstrap.cc:492] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:19.538507 5654 consensus_peers.cc:485] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08aac6e88b7c41d8a59d1e12e7f6c14e. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.539846 5841 ts_tablet_manager.cc:1888] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.541108 5841 log.cc:1196] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/385f4c1eb7f4493a924b813734ddcfdb
I20240507 08:12:19.542495 5841 ts_tablet_manager.cc:1909] T 385f4c1eb7f4493a924b813734ddcfdb P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:19.540505 5442 raft_consensus.cc:2454] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:19.544824 5656 leader_election.cc:302] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.545284 7358 raft_consensus.cc:2781] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:19.545478 7358 raft_consensus.cc:686] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:19.545861 7358 consensus_queue.cc:237] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.546416 5840 tablet_bootstrap.cc:492] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.546581 5840 ts_tablet_manager.cc:1367] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20240507 08:12:19.539889 5843 tablet_bootstrap.cc:654] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.547333 5840 raft_consensus.cc:348] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.547546 5840 raft_consensus.cc:374] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.547653 5840 raft_consensus.cc:729] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.548036 5840 consensus_queue.cc:260] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.548663 5840 ts_tablet_manager.cc:1398] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:19.548970 5840 tablet_bootstrap.cc:492] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.550613 5840 tablet_bootstrap.cc:654] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.554890 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_015ef8e34abd4318ac03fb75c3e931a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.555527 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_015ef8e34abd4318ac03fb75c3e931a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.556915 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c6cb80106e7a43168975c570c122e5e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.557387 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6cb80106e7a43168975c570c122e5e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.557968 5839 tablet_bootstrap.cc:492] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.557974 5256 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:19.558215 5839 ts_tablet_manager.cc:1367] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.261s user 0.016s sys 0.002s
I20240507 08:12:19.558933 5839 raft_consensus.cc:348] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.559190 5839 raft_consensus.cc:374] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.559301 5839 raft_consensus.cc:729] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.559598 5839 consensus_queue.cc:260] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.560258 5839 ts_tablet_manager.cc:1398] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.560557 5839 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.561714 5843 tablet_bootstrap.cc:492] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.561856 5843 ts_tablet_manager.cc:1367] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.001s
I20240507 08:12:19.562753 5843 raft_consensus.cc:348] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.562937 5843 raft_consensus.cc:374] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.562999 5843 raft_consensus.cc:729] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.566104 5843 consensus_queue.cc:260] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.570040 5843 ts_tablet_manager.cc:1398] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240507 08:12:19.571748 6513 ts_tablet_manager.cc:1888] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:19.572237 6513 log.cc:1196] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/975f22194aff40609a3cfa63b9b3aca2
I20240507 08:12:19.558450 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f59032e865db4953a0e6c4517ae795ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.573879 6513 ts_tablet_manager.cc:1909] T 975f22194aff40609a3cfa63b9b3aca2 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:19.574095 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f59032e865db4953a0e6c4517ae795ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.575310 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1190) took 3901 ms. Trace:
I20240507 08:12:19.578979 5839 tablet_bootstrap.cc:654] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.573436 5840 tablet_bootstrap.cc:492] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.579223 5840 ts_tablet_manager.cc:1367] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20240507 08:12:19.580029 5840 raft_consensus.cc:348] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.580281 5840 raft_consensus.cc:374] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.580384 5840 raft_consensus.cc:729] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.580664 5840 consensus_queue.cc:260] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.581221 5840 ts_tablet_manager.cc:1398] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.581640 5840 tablet_bootstrap.cc:492] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.583210 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_512dbcaed1c74258922227c3af764ede"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.583814 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_512dbcaed1c74258922227c3af764ede in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.584604 5843 tablet_bootstrap.cc:492] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.585088 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a20a31ad7635469c8353b7e7759c0a39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.585613 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a20a31ad7635469c8353b7e7759c0a39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.586990 5840 tablet_bootstrap.cc:654] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.598091 7378 raft_consensus.cc:482] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.598249 7378 raft_consensus.cc:504] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.598881 7378 leader_election.cc:288] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.601060 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:19.601603 5377 leader_election.cc:341] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.601605 6513 rpcz_store.cc:276] 0507 08:12:15.673926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.675009 (+ 1083us) service_pool.cc:225] Handling call
0507 08:12:19.535434 (+3860425us) ts_tablet_manager.cc:1120] Deleting tablet 975f22194aff40609a3cfa63b9b3aca2
0507 08:12:19.535442 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.554148 (+ 18706us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.571621 (+ 17473us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.575292 (+ 3671us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3858507}
I20240507 08:12:19.602119 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 975f22194aff40609a3cfa63b9b3aca2 (table default.Table_8493d14dd0f8480591e6e0a0196c77cb [id=3b2ccfca7fae432bb6acfa7e375f6ebe]) successfully deleted
I20240507 08:12:19.602648 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1192) took 3870 ms. Trace:
I20240507 08:12:19.602782 6513 rpcz_store.cc:276] 0507 08:12:15.732440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.732488 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:19.602524 (+3870036us) ts_tablet_manager.cc:1120] Deleting tablet f9b60e7fd01c4c96911fff6bf62ade96
0507 08:12:19.602534 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.602627 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3869878}
W20240507 08:12:19.603286 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet f9b60e7fd01c4c96911fff6bf62ade96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f9b60e7fd01c4c96911fff6bf62ade96 already in progress: creating tablet
I20240507 08:12:19.603688 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1191) took 3871 ms. Trace:
I20240507 08:12:19.603804 6513 rpcz_store.cc:276] 0507 08:12:15.732325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.733468 (+ 1143us) service_pool.cc:225] Handling call
0507 08:12:19.603580 (+3870112us) ts_tablet_manager.cc:1120] Deleting tablet ddcb720c09c5486995391837fc54e7ce
0507 08:12:19.603589 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.603670 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3869951}
W20240507 08:12:19.604322 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet ddcb720c09c5486995391837fc54e7ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ddcb720c09c5486995391837fc54e7ce already in progress: creating tablet
I20240507 08:12:19.604812 6513 tablet_replica.cc:330] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.605063 6513 raft_consensus.cc:2229] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:19.605429 6513 raft_consensus.cc:2258] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.613375 6513 ts_tablet_manager.cc:1875] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.580404 7079 ts_tablet_manager.cc:1888] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.613682 7079 log.cc:1196] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/049483b0533642f2b46d8fa0e9b123cc
I20240507 08:12:19.614113 7079 ts_tablet_manager.cc:1909] T 049483b0533642f2b46d8fa0e9b123cc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:19.614461 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1210) took 2347 ms. Trace:
I20240507 08:12:19.614585 7079 rpcz_store.cc:276] 0507 08:12:17.267490 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.267561 (+ 71us) service_pool.cc:225] Handling call
0507 08:12:19.437506 (+2169945us) ts_tablet_manager.cc:1120] Deleting tablet 049483b0533642f2b46d8fa0e9b123cc
0507 08:12:19.437544 (+ 38us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.490872 (+ 53328us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.580300 (+ 89428us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.614444 (+ 34144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169740}
I20240507 08:12:19.615072 7079 tablet_replica.cc:330] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:19.600011 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:19.615636 5378 leader_election.cc:341] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.615752 5378 leader_election.cc:302] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.615885 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 049483b0533642f2b46d8fa0e9b123cc (table default.Table_fcf86fbc38ac4137b5b9acb214330443 [id=37f8775ed9d94ba58b5fd966ce2b04c5]) successfully deleted
I20240507 08:12:19.616148 7378 raft_consensus.cc:2726] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.613426 5840 tablet_bootstrap.cc:492] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.617717 5840 ts_tablet_manager.cc:1367] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20240507 08:12:19.618505 5840 raft_consensus.cc:348] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.618779 5840 raft_consensus.cc:374] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.618868 5840 raft_consensus.cc:729] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.619103 5840 consensus_queue.cc:260] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.619787 5840 ts_tablet_manager.cc:1398] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.620062 5840 tablet_bootstrap.cc:492] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.620373 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_99fc6e6505b14088ab7e1344ee923296"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.620934 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99fc6e6505b14088ab7e1344ee923296 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.621819 5840 tablet_bootstrap.cc:654] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.620930 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7420d3e805d64c9096c3e5411ad789db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.622943 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7420d3e805d64c9096c3e5411ad789db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.625057 7602 raft_consensus.cc:482] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.625182 7602 raft_consensus.cc:504] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.625936 7602 leader_election.cc:288] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.626789 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3d66e54ff28e4b2799cb6391b6aba67f" } new_table_name: "default.Table_31f616da606e4df081b75ae47951e6b2" modify_external_catalogs: true
W20240507 08:12:19.626909 5517 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 957775c01be94cc081db344c95a81310
W20240507 08:12:19.627106 5520 leader_election.cc:341] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 957775c01be94cc081db344c95a81310
I20240507 08:12:19.627184 5520 leader_election.cc:302] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:19.627631 7602 raft_consensus.cc:2726] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.631069 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_60d63759e9ab452b86714212d928185f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.631376 7378 raft_consensus.cc:482] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:19.631537 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60d63759e9ab452b86714212d928185f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.631498 7378 raft_consensus.cc:504] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.632197 7378 leader_election.cc:288] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:19.633260 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee9e434083f4fa6859cfe2b7cca16e5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:19.633756 5378 leader_election.cc:341] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.634135 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee9e434083f4fa6859cfe2b7cca16e5" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
W20240507 08:12:19.634548 5377 leader_election.cc:341] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.634649 5377 leader_election.cc:302] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.635257 7499 raft_consensus.cc:2726] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.640506 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c39926d459d14c72884a82bf549db3ff" } new_table_name: "default.Table_db93bd321ad74ca69a31a797c85f6030" modify_external_catalogs: true
I20240507 08:12:19.642033 6513 ts_tablet_manager.cc:1888] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.645367 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_908d49dcd5814349aee9767384232deb" } modify_external_catalogs: true
I20240507 08:12:19.647049 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_908d49dcd5814349aee9767384232deb" } modify_external_catalogs: true
I20240507 08:12:19.642516 5840 tablet_bootstrap.cc:492] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.648371 5840 ts_tablet_manager.cc:1367] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
I20240507 08:12:19.649091 5840 raft_consensus.cc:348] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.649279 5840 raft_consensus.cc:374] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.649336 5840 raft_consensus.cc:729] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.649513 5840 consensus_queue.cc:260] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.650202 5840 ts_tablet_manager.cc:1398] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.650452 5840 tablet_bootstrap.cc:492] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.651711 7079 raft_consensus.cc:2229] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.651882 7079 raft_consensus.cc:2258] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.651949 5840 tablet_bootstrap.cc:654] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.652434 7079 ts_tablet_manager.cc:1875] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.655730 5843 tablet_bootstrap.cc:654] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.661042 6513 log.cc:1196] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b8ebbcf0d5a54d63a96a9374e839af83
I20240507 08:12:19.661746 6513 ts_tablet_manager.cc:1909] T b8ebbcf0d5a54d63a96a9374e839af83 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.666776 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1200) took 3921 ms. Trace:
I20240507 08:12:19.578284 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1219) took 3030 ms. Trace:
I20240507 08:12:19.668592 6513 rpcz_store.cc:276] 0507 08:12:15.745198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.749616 (+ 4418us) service_pool.cc:225] Handling call
0507 08:12:19.604745 (+3855129us) ts_tablet_manager.cc:1120] Deleting tablet b8ebbcf0d5a54d63a96a9374e839af83
0507 08:12:19.604754 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.623823 (+ 19069us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.641927 (+ 18104us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.666757 (+ 24830us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3854961}
I20240507 08:12:19.669462 6513 tablet_replica.cc:330] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.669593 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b8ebbcf0d5a54d63a96a9374e839af83 (table default.Table_0f7557ddeec745f59ac90d4085a9015c [id=2a51124d3ab54fc58394e23a6aead27c]) successfully deleted
I20240507 08:12:19.671540 5841 rpcz_store.cc:276] 0507 08:12:16.547304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.547491 (+ 187us) service_pool.cc:225] Handling call
0507 08:12:19.488203 (+2940712us) ts_tablet_manager.cc:1120] Deleting tablet 385f4c1eb7f4493a924b813734ddcfdb
0507 08:12:19.488211 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.523085 (+ 34874us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.539728 (+ 16643us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.577956 (+ 38228us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2940573}
I20240507 08:12:19.672183 6513 raft_consensus.cc:2229] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.672335 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 385f4c1eb7f4493a924b813734ddcfdb (table default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 [id=a286976f28f7468784296bf1ada7bd29]) successfully deleted
I20240507 08:12:19.672405 6513 raft_consensus.cc:2258] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.672917 6513 ts_tablet_manager.cc:1875] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.673071 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1220) took 3125 ms. Trace:
I20240507 08:12:19.673206 5841 rpcz_store.cc:276] 0507 08:12:16.547444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.547664 (+ 220us) service_pool.cc:225] Handling call
0507 08:12:19.672975 (+3125311us) ts_tablet_manager.cc:1120] Deleting tablet ba7c1aaf55d74c979ec1fc0a57f15f44
0507 08:12:19.672984 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.673054 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3125173}
W20240507 08:12:19.674618 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ba7c1aaf55d74c979ec1fc0a57f15f44 because the tablet was not found. No further retry: Not found: Tablet not found: ba7c1aaf55d74c979ec1fc0a57f15f44
I20240507 08:12:19.674333 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1218) took 3127 ms. Trace:
I20240507 08:12:19.674880 5841 rpcz_store.cc:276] 0507 08:12:16.547200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.547818 (+ 618us) service_pool.cc:225] Handling call
0507 08:12:19.674252 (+3126434us) ts_tablet_manager.cc:1120] Deleting tablet 831153d2b89e43f8b08b086f7a8f3a54
0507 08:12:19.674261 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.674318 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3126305}
W20240507 08:12:19.675269 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 831153d2b89e43f8b08b086f7a8f3a54 because the tablet was not found. No further retry: Not found: Tablet not found: 831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:12:19.675320 5841 tablet_replica.cc:330] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:19.678330 5841 raft_consensus.cc:2229] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.678550 5841 raft_consensus.cc:2258] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.680275 5841 ts_tablet_manager.cc:1875] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.686353 5839 tablet_bootstrap.cc:492] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.689006 5843 tablet_bootstrap.cc:492] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.689191 5843 ts_tablet_manager.cc:1367] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.105s user 0.002s sys 0.002s
I20240507 08:12:19.690057 5843 raft_consensus.cc:348] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.690318 5843 raft_consensus.cc:374] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.690343 5840 tablet_bootstrap.cc:492] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.690441 5843 raft_consensus.cc:729] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.690462 5840 ts_tablet_manager.cc:1367] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.008s
I20240507 08:12:19.690721 5843 consensus_queue.cc:260] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.691150 5840 raft_consensus.cc:348] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.691279 5843 ts_tablet_manager.cc:1398] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:19.691401 5840 raft_consensus.cc:374] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.691541 5843 tablet_bootstrap.cc:492] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.691546 5840 raft_consensus.cc:729] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.691763 5840 consensus_queue.cc:260] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.692548 5840 ts_tablet_manager.cc:1398] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:19.693138 5843 tablet_bootstrap.cc:654] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.693233 5840 tablet_bootstrap.cc:492] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.694789 5840 tablet_bootstrap.cc:654] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.694979 5839 ts_tablet_manager.cc:1367] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.134s user 0.005s sys 0.009s
I20240507 08:12:19.702281 5839 raft_consensus.cc:348] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.703351 5839 raft_consensus.cc:374] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.703480 5839 raft_consensus.cc:729] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.704203 5839 consensus_queue.cc:260] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:19.706858 5250 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet 37a23d112fcd4ea1b0c2f86133e82e32 (table default.Table_af114d78455749ce9330a056f6505e04 [id=513cac3f33c8405183c2af4f13522541], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 37a23d112fcd4ea1b0c2f86133e82e32 consensus state has no leader
I20240507 08:12:19.709061 5839 ts_tablet_manager.cc:1398] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.000s sys 0.001s
I20240507 08:12:19.714959 5839 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.716486 5839 tablet_bootstrap.cc:654] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.711689 6513 ts_tablet_manager.cc:1888] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.723217 6513 log.cc:1196] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/14929efb0ed049d6b554598a5b8cb494
I20240507 08:12:19.723664 6513 ts_tablet_manager.cc:1909] T 14929efb0ed049d6b554598a5b8cb494 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.724056 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1199) took 3978 ms. Trace:
I20240507 08:12:19.724192 6513 rpcz_store.cc:276] 0507 08:12:15.745119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.749617 (+ 4498us) service_pool.cc:225] Handling call
0507 08:12:19.669417 (+3919800us) ts_tablet_manager.cc:1120] Deleting tablet 14929efb0ed049d6b554598a5b8cb494
0507 08:12:19.669426 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.679029 (+ 9603us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.711569 (+ 32540us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.724033 (+ 12464us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2040,"tablet-delete.queue_time_us":3919366}
I20240507 08:12:19.724702 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1201) took 3962 ms. Trace:
I20240507 08:12:19.724694 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 14929efb0ed049d6b554598a5b8cb494 (table default.Table_b21b58e8d49943268e7491117d5d717e [id=afe7e0cf53aa444f84f4773519b48c85]) successfully deleted
I20240507 08:12:19.724902 6513 rpcz_store.cc:276] 0507 08:12:15.762028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.762081 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:19.724588 (+3962507us) ts_tablet_manager.cc:1120] Deleting tablet 38fc0180e9d34d11885b819529407930
0507 08:12:19.724596 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.724688 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3962313}
I20240507 08:12:19.725258 6513 tablet_replica.cc:330] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.725486 6513 raft_consensus.cc:2229] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:19.725787 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 38fc0180e9d34d11885b819529407930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38fc0180e9d34d11885b819529407930 already in progress: creating tablet
I20240507 08:12:19.724762 5840 tablet_bootstrap.cc:492] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.729662 6513 raft_consensus.cc:2258] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.729772 5840 ts_tablet_manager.cc:1367] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240507 08:12:19.730309 6513 ts_tablet_manager.cc:1875] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.730779 5840 raft_consensus.cc:348] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.730952 5840 raft_consensus.cc:374] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.731010 5840 raft_consensus.cc:729] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.731205 5840 consensus_queue.cc:260] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.731757 5840 ts_tablet_manager.cc:1398] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.732028 5840 tablet_bootstrap.cc:492] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.733433 5840 tablet_bootstrap.cc:654] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.735216 5839 tablet_bootstrap.cc:492] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.735448 5839 ts_tablet_manager.cc:1367] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.003s
I20240507 08:12:19.736142 5839 raft_consensus.cc:348] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.736449 5839 raft_consensus.cc:374] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.736541 5839 raft_consensus.cc:729] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.736833 5839 consensus_queue.cc:260] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.738951 7079 ts_tablet_manager.cc:1888] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.741981 7079 log.cc:1196] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/aa094773f3bf418492bcd5b5021fd429
I20240507 08:12:19.744133 5839 ts_tablet_manager.cc:1398] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240507 08:12:19.743759 7079 ts_tablet_manager.cc:1909] T aa094773f3bf418492bcd5b5021fd429 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:19.750078 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 98b59d5840e04d05b98a6fca7146db20 (DEFAULT_TABLE table=default.Table_344c146019d84e2f8e80331a5530bccf [id=7e16a28e6b15449ca3fdfc891e211580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.752195 5841 ts_tablet_manager.cc:1888] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.757370 5839 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.759470 5839 tablet_bootstrap.cc:654] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.760815 5840 tablet_bootstrap.cc:492] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.760923 5840 ts_tablet_manager.cc:1367] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.004s
I20240507 08:12:19.761555 5840 raft_consensus.cc:348] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.761757 5840 raft_consensus.cc:374] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.761801 5840 raft_consensus.cc:729] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.761965 5840 consensus_queue.cc:260] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.762408 5840 ts_tablet_manager.cc:1398] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
W20240507 08:12:19.762535 5633 tablet.cc:2369] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:12:19.762666 5840 tablet_bootstrap.cc:492] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.764027 5840 tablet_bootstrap.cc:654] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.767398 6513 ts_tablet_manager.cc:1888] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.773958 6513 log.cc:1196] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:12:19.775446 5259 master_service.cc:946] ReplaceTablet: received request to replace tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:19.776127 5839 tablet_bootstrap.cc:492] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.776340 5839 ts_tablet_manager.cc:1367] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
W20240507 08:12:19.776612 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65f5a93e7b95418bb61e6adfb9ca78d0 consensus state has no leader
I20240507 08:12:19.777158 5839 raft_consensus.cc:348] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.777423 5839 raft_consensus.cc:374] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.777531 5839 raft_consensus.cc:729] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.777884 5839 consensus_queue.cc:260] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.778905 5840 tablet_bootstrap.cc:492] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
W20240507 08:12:19.779022 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8adad098c934d11982d39aa1cf8b4bc consensus state has no leader
I20240507 08:12:19.779448 5840 ts_tablet_manager.cc:1367] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240507 08:12:19.779707 5262 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_f7583788c28a4279a8cd5d8684eb4393 [id=aac50aefe6524a84b1e4564d3f0ce394] alter complete (version 4)
I20240507 08:12:19.780083 5840 raft_consensus.cc:348] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.780248 5840 raft_consensus.cc:374] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.780297 5840 raft_consensus.cc:729] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.780345 5839 ts_tablet_manager.cc:1398] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240507 08:12:19.780635 6513 ts_tablet_manager.cc:1909] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.780521 5840 consensus_queue.cc:260] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.781000 5840 ts_tablet_manager.cc:1398] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240507 08:12:19.781138 5262 master_service.cc:946] ReplaceTablet: received request to replace tablet cbf7dd6413fb47108cce25cde80738fe from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:19.781226 5840 tablet_bootstrap.cc:492] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.781349 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1206) took 4015 ms. Trace:
I20240507 08:12:19.781411 6513 rpcz_store.cc:276] 0507 08:12:15.766187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.766246 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:19.725217 (+3958971us) ts_tablet_manager.cc:1120] Deleting tablet 84212d6cb2c742c1a0f3fdae8be8ebe1
0507 08:12:19.725226 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.746793 (+ 21567us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.767302 (+ 20509us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.781333 (+ 14031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3958795}
I20240507 08:12:19.781814 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 (table default.Table_54a33e3ca40248028587733beb4f6743 [id=9b06744f353d42a69ea7f5b5b60f3a52]) successfully deleted
I20240507 08:12:19.783821 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_454ae1a1af424a77a79ea0010b86a0bb" } modify_external_catalogs: true
I20240507 08:12:19.783980 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_454ae1a1af424a77a79ea0010b86a0bb" } modify_external_catalogs: true
I20240507 08:12:19.784116 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1207) took 4017 ms. Trace:
I20240507 08:12:19.784188 6513 rpcz_store.cc:276] 0507 08:12:15.766300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.766451 (+ 151us) service_pool.cc:225] Handling call
0507 08:12:19.784021 (+4017570us) ts_tablet_manager.cc:1120] Deleting tablet b6e68364238c4b66b8c0de0f4a2ea855
0507 08:12:19.784030 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.784100 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4017452}
W20240507 08:12:19.785710 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b6e68364238c4b66b8c0de0f4a2ea855 because the tablet was not found. No further retry: Not found: Tablet not found: b6e68364238c4b66b8c0de0f4a2ea855
I20240507 08:12:19.786373 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a34af5c61e5649e5abc9c88235891674" } new_table_name: "default.Table_bb514540e69b47f0870bc08ca4974547" modify_external_catalogs: true
I20240507 08:12:19.786808 6513 tablet_replica.cc:330] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.786795 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet cd4379b6fded4a52843f55f27ae67121 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.786821 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2c24bc292e0e44fba3e26850b6615d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.786988 6513 raft_consensus.cc:2229] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.787133 6513 raft_consensus.cc:2258] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.787184 5840 tablet_bootstrap.cc:654] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.787637 6513 ts_tablet_manager.cc:1875] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.790175 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ae198ba314f04af0ba8dde8c1c3f7839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.790817 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae198ba314f04af0ba8dde8c1c3f7839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.791599 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 975f22194aff40609a3cfa63b9b3aca2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:15 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:19.792469 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c24bc292e0e44fba3e26850b6615d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.792644 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e37b73134cf42ad84b6eec90c9b805f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.792862 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.793195 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.796489 5840 tablet_bootstrap.cc:492] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.796658 5840 ts_tablet_manager.cc:1367] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20240507 08:12:19.797052 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 3ccc3e67eeb34c42b36edde7bacd23a5 with delete_type TABLET_DATA_DELETED (replaced by tablet f89efb8414dc49369d361adf908927e6) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.797271 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.797498 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 3fce98f472ef46589dbea6b6aefa6572 with delete_type TABLET_DATA_DELETED (replaced by tablet bf8fe5157dd9491cac7e0237c3d39ec1) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.797420 5840 raft_consensus.cc:348] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.797605 5840 raft_consensus.cc:374] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.797684 5840 raft_consensus.cc:729] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.797871 5840 consensus_queue.cc:260] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.798388 5840 ts_tablet_manager.cc:1398] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:19.801327 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b59d5840e04d05b98a6fca7146db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.802598 5840 tablet_bootstrap.cc:492] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.804147 5840 tablet_bootstrap.cc:654] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.806354 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 4bd0f775bfa344b9b02afc875db46af7
I20240507 08:12:19.807533 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.807873 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 669ffac4b4f34551ba930f0a259e78a6
I20240507 08:12:19.808493 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.808737 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1211) took 2481 ms. Trace:
I20240507 08:12:19.808768 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.808815 7079 rpcz_store.cc:276] 0507 08:12:17.327553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.327604 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:19.615032 (+2287428us) ts_tablet_manager.cc:1120] Deleting tablet aa094773f3bf418492bcd5b5021fd429
0507 08:12:19.615039 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.700592 (+ 85553us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.738368 (+ 37776us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.808720 (+ 70352us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2287254}
I20240507 08:12:19.809262 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1209) took 2559 ms. Trace:
I20240507 08:12:19.809335 7079 rpcz_store.cc:276] 0507 08:12:17.250249 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.332068 (+ 81819us) service_pool.cc:225] Handling call
0507 08:12:19.809143 (+2477075us) ts_tablet_manager.cc:1120] Deleting tablet fda4b1ceaf0b4f3385258f3703e4461b
0507 08:12:19.809152 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.809245 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2476903}
I20240507 08:12:19.809638 5839 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.809671 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 6e2759ebfa0840f8b21d749354768c8d
I20240507 08:12:19.810392 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2759ebfa0840f8b21d749354768c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.809660 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1212) took 2418 ms. Trace:
I20240507 08:12:19.811389 7079 rpcz_store.cc:276] 0507 08:12:17.390923 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.390984 (+ 61us) service_pool.cc:225] Handling call
0507 08:12:19.809543 (+2418559us) ts_tablet_manager.cc:1120] Deleting tablet 3eb49743134b4f84aa6fdd9760c13504
0507 08:12:19.809551 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.809644 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2418365}
I20240507 08:12:19.812036 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1213) took 2419 ms. Trace:
I20240507 08:12:19.812126 7079 rpcz_store.cc:276] 0507 08:12:17.392148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.392199 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:19.811803 (+2419604us) ts_tablet_manager.cc:1120] Deleting tablet 5481ddbcd31449c4b44afb87a97138bd
0507 08:12:19.811815 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.812016 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2419452}
I20240507 08:12:19.812548 7656 raft_consensus.cc:482] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.812680 7656 raft_consensus.cc:504] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.813155 7079 tablet_replica.cc:330] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:19.813282 7656 leader_election.cc:288] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.813335 7079 raft_consensus.cc:2229] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:19.813643 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 411619d6fb934f63b059ae7a489c8bf8 consensus state has no leader
I20240507 08:12:19.814337 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet aa094773f3bf418492bcd5b5021fd429 (table default.Table_610ce2d36eaf4152b4c056012a719c18 [id=219dcab59f1141b0879949dbd03acf35]) successfully deleted
I20240507 08:12:19.814687 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 92e1e4cbc3e34af8bb56fc0ae41b7377
I20240507 08:12:19.814878 5839 tablet_bootstrap.cc:654] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.814846 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2759ebfa0840f8b21d749354768c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.814868 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:19.815836 5654 leader_election.cc:341] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.817003 7602 raft_consensus.cc:482] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.817162 7602 raft_consensus.cc:504] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.817941 7602 leader_election.cc:288] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.818655 5264 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4638b813c41e48b4b3d7afef76128b8a" } modify_external_catalogs: true
W20240507 08:12:19.819092 5520 leader_election.cc:341] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 8bb62a72ed2248f6886823b9c64ff4c6
I20240507 08:12:19.819674 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_939ac1a3eef54238b67882f7284a2a94" } new_table_name: "default.Table_93c680c498364e56a9dbc2cb87da079f" modify_external_catalogs: true
I20240507 08:12:19.819922 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb62a72ed2248f6886823b9c64ff4c6" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:19.813475 7079 raft_consensus.cc:2258] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.820394 5517 leader_election.cc:302] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:19.820650 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_46e3bcdbfbbe464ab9b2e5ac327883eb" } new_table_name: "default.Table_fe1dc5df13924e7988b0a9751ed2979a" modify_external_catalogs: true
W20240507 08:12:19.814548 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet fda4b1ceaf0b4f3385258f3703e4461b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fda4b1ceaf0b4f3385258f3703e4461b already in progress: creating tablet
I20240507 08:12:19.821563 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:19.821750 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3eb49743134b4f84aa6fdd9760c13504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3eb49743134b4f84aa6fdd9760c13504 already in progress: creating tablet
W20240507 08:12:19.821959 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5481ddbcd31449c4b44afb87a97138bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5481ddbcd31449c4b44afb87a97138bd already in progress: creating tablet
I20240507 08:12:19.822304 7602 raft_consensus.cc:2726] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:19.822340 5259 catalog_manager.cc:6554] ReplaceTablet: tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 deleted and replaced by tablet 89df9dcebed34d7da51b07b8e43880be
I20240507 08:12:19.822580 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 with delete_type TABLET_DATA_DELETED (replaced by tablet 89df9dcebed34d7da51b07b8e43880be) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.822649 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c60abdd82927451d901f94b4868f5c19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.823160 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c60abdd82927451d901f94b4868f5c19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.823789 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 with delete_type TABLET_DATA_DELETED (replaced by tablet 89df9dcebed34d7da51b07b8e43880be) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.825790 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_85ac61888c9941a195772caf2024259f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.825876 5656 consensus_peers.cc:485] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8bb62a72ed2248f6886823b9c64ff4c6. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.826669 7079 ts_tablet_manager.cc:1875] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.827713 5264 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4638b813c41e48b4b3d7afef76128b8a" } modify_external_catalogs: true
W20240507 08:12:19.831708 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85ac61888c9941a195772caf2024259f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.836410 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet cbf7dd6413fb47108cce25cde80738fe
I20240507 08:12:19.836892 5262 catalog_manager.cc:6554] ReplaceTablet: tablet cbf7dd6413fb47108cce25cde80738fe deleted and replaced by tablet c6130da2ee3e4ecbbe13fcfae98a4953
I20240507 08:12:19.837152 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf7dd6413fb47108cce25cde80738fe with delete_type TABLET_DATA_DELETED (replaced by tablet c6130da2ee3e4ecbbe13fcfae98a4953) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.837193 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_06265a021c464ab0b3391f95db5c2354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.837668 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06265a021c464ab0b3391f95db5c2354 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.839709 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf7dd6413fb47108cce25cde80738fe with delete_type TABLET_DATA_DELETED (replaced by tablet c6130da2ee3e4ecbbe13fcfae98a4953) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.841462 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 33875331045146a9a02559a0b37d5c6f with delete_type TABLET_DATA_DELETED (replaced by tablet 4ac55233449844d2bd40fee0be427e13) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.841362 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.841662 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 831153d2b89e43f8b08b086f7a8f3a54 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:19.841862 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.842232 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb62a72ed2248f6886823b9c64ff4c6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.842180 7656 consensus_queue.cc:1035] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.844763 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet d0eed609cb24474592c5b66b63e7669f
I20240507 08:12:19.845384 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet d0eed609cb24474592c5b66b63e7669f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.846011 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_74a33830b6b843e6b28ba70ec7b0d05d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.846462 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74a33830b6b843e6b28ba70ec7b0d05d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.847414 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet d0eed609cb24474592c5b66b63e7669f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.835093 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 449d6c8270184ce2918cee8ce731f788 (table default.Table_2c4d993ae928415181f53571b6bf0815 [id=3764a23f5a7847c8b46385bdfaf51f20]). Expected version 1 got 0
I20240507 08:12:19.850359 5256 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 66247e08dfdf47f69ba24b074c8080f0 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]). Expected version 2 got 0
I20240507 08:12:19.851723 7079 ts_tablet_manager.cc:1888] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.857303 6513 ts_tablet_manager.cc:1888] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:19.856865 5839 tablet_bootstrap.cc:492] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:19.857651 5839 ts_tablet_manager.cc:1367] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20240507 08:12:19.858588 5839 raft_consensus.cc:348] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.858908 5839 raft_consensus.cc:374] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.859019 5839 raft_consensus.cc:729] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.859292 5839 consensus_queue.cc:260] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.862001 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 1314775fa5df438ab3aa1eb0075e7f78 (DEFAULT_TABLE table=default.Table_697e51662b0b4accadb633f6980acf76 [id=166134823162479aa6e5354d581c6f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.862366 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1314775fa5df438ab3aa1eb0075e7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.864446 5839 ts_tablet_manager.cc:1398] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240507 08:12:19.864699 5839 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:19.865974 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 26d09d06099547d39a016d38c14d4ca7 (DEFAULT_TABLE table=default.Table_fb68d3d3d5c842b0a95906db237261d3 [id=dd50861c54af446b8cad51ad68e2df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.866202 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 7b13eeb171b743b8b831d214d400ee7b (DEFAULT_TABLE table=default.Table_6362b6a3832a42099cbf4fc493c068fe [id=c097b927dfea4c698592f003be792282]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.866308 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d09d06099547d39a016d38c14d4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.866514 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b13eeb171b743b8b831d214d400ee7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.866777 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 915ac28ed118432f9a02f417226f2ecc (DEFAULT_TABLE table=default.Table_a3550d2ca3f04d659cd01f9eb838edc1 [id=5e7e94e86d0b4a63ac81fbe3412fcb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.867089 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915ac28ed118432f9a02f417226f2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.875741 7079 log.cc:1196] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/34adee00b529424b9f274c61ce73b0bc
I20240507 08:12:19.785773 5427 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.786065 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet eec3704b135740f1a906d60f3ea87ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.786217 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b13fbddcab5460db69f43e63f490fba with delete_type TABLET_DATA_DELETED (replaced by tablet b7fe4513895d4f5d9ca8d114eb4a827f) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.813918 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:19.877879 5447 raft_consensus.cc:2454] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:19.878000 5841 log.cc:1196] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e14487e625374060b118c9cce4500b3f
I20240507 08:12:19.819213 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e2759ebfa0840f8b21d749354768c8d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.820278 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 669ffac4b4f34551ba930f0a259e78a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.821136 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bd0f775bfa344b9b02afc875db46af7 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.823709 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 with delete_type TABLET_DATA_DELETED (replaced by tablet 89df9dcebed34d7da51b07b8e43880be) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.839097 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet cbf7dd6413fb47108cce25cde80738fe with delete_type TABLET_DATA_DELETED (replaced by tablet c6130da2ee3e4ecbbe13fcfae98a4953) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.845484 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet d0eed609cb24474592c5b66b63e7669f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.860846 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 98b59d5840e04d05b98a6fca7146db20 (DEFAULT_TABLE table=default.Table_344c146019d84e2f8e80331a5530bccf [id=7e16a28e6b15449ca3fdfc891e211580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.861866 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 1314775fa5df438ab3aa1eb0075e7f78 (DEFAULT_TABLE table=default.Table_697e51662b0b4accadb633f6980acf76 [id=166134823162479aa6e5354d581c6f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.863433 5408 tablet_service.cc:1467] Processing CreateTablet for tablet 26d09d06099547d39a016d38c14d4ca7 (DEFAULT_TABLE table=default.Table_fb68d3d3d5c842b0a95906db237261d3 [id=dd50861c54af446b8cad51ad68e2df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.864939 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 915ac28ed118432f9a02f417226f2ecc (DEFAULT_TABLE table=default.Table_a3550d2ca3f04d659cd01f9eb838edc1 [id=5e7e94e86d0b4a63ac81fbe3412fcb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.866297 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 7b13eeb171b743b8b831d214d400ee7b (DEFAULT_TABLE table=default.Table_6362b6a3832a42099cbf4fc493c068fe [id=c097b927dfea4c698592f003be792282]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.878610 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b59d5840e04d05b98a6fca7146db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.878746 5841 ts_tablet_manager.cc:1909] T e14487e625374060b118c9cce4500b3f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:19.879102 7079 ts_tablet_manager.cc:1909] T 34adee00b529424b9f274c61ce73b0bc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:19.862370 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 98b59d5840e04d05b98a6fca7146db20 (DEFAULT_TABLE table=default.Table_344c146019d84e2f8e80331a5530bccf [id=7e16a28e6b15449ca3fdfc891e211580]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.879616 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98b59d5840e04d05b98a6fca7146db20. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.862880 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 1314775fa5df438ab3aa1eb0075e7f78 (DEFAULT_TABLE table=default.Table_697e51662b0b4accadb633f6980acf76 [id=166134823162479aa6e5354d581c6f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.882016 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1314775fa5df438ab3aa1eb0075e7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.884658 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d09d06099547d39a016d38c14d4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.864230 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 26d09d06099547d39a016d38c14d4ca7 (DEFAULT_TABLE table=default.Table_fb68d3d3d5c842b0a95906db237261d3 [id=dd50861c54af446b8cad51ad68e2df57]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.885087 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26d09d06099547d39a016d38c14d4ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.885138 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b13eeb171b743b8b831d214d400ee7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.885468 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915ac28ed118432f9a02f417226f2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.886178 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1221) took 3337 ms. Trace:
I20240507 08:12:19.886253 5841 rpcz_store.cc:276] 0507 08:12:16.548553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.548684 (+ 131us) service_pool.cc:225] Handling call
0507 08:12:19.675276 (+3126592us) ts_tablet_manager.cc:1120] Deleting tablet e14487e625374060b118c9cce4500b3f
0507 08:12:19.675286 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.703224 (+ 27938us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.752097 (+ 48873us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.886160 (+134063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3126449}
I20240507 08:12:19.886693 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e14487e625374060b118c9cce4500b3f (table default.Table_5efeda526693400790a6dfb0d118fac0 [id=8adafd6d08374df7a4f0fa1fd90991fe]) successfully deleted
I20240507 08:12:19.886979 5841 tablet_replica.cc:330] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:19.887142 5841 raft_consensus.cc:2229] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.887300 5841 raft_consensus.cc:2258] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.889762 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1314775fa5df438ab3aa1eb0075e7f78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.891675 5843 tablet_bootstrap.cc:492] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:19.891790 5843 ts_tablet_manager.cc:1367] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.200s user 0.003s sys 0.002s
I20240507 08:12:19.892462 5843 raft_consensus.cc:348] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.892652 5843 raft_consensus.cc:374] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.892700 5843 raft_consensus.cc:729] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.892902 5843 consensus_queue.cc:260] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.893390 5843 ts_tablet_manager.cc:1398] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:19.871109 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 915ac28ed118432f9a02f417226f2ecc (DEFAULT_TABLE table=default.Table_a3550d2ca3f04d659cd01f9eb838edc1 [id=5e7e94e86d0b4a63ac81fbe3412fcb5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.894212 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 915ac28ed118432f9a02f417226f2ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.896713 5843 tablet_bootstrap.cc:492] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:19.899909 5417 tablet_service.cc:1467] Processing CreateTablet for tablet b0aaf4f1ff1841d6bd3eb9df30c39967 (DEFAULT_TABLE table=default.Table_896f00e31db14579b52336dddabe10ec [id=40c6022946364cc8968da1751275c8b3]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:19.871917 5654 consensus_peers.cc:485] T c3ed7d7d2e8e424c90048c87c0fb0ca6 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.878428 5656 leader_election.cc:302] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:19.901206 7649 raft_consensus.cc:2781] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:19.901319 7649 raft_consensus.cc:482] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:19.901397 7649 raft_consensus.cc:3037] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.904120 7649 raft_consensus.cc:504] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:19.904922 7649 leader_election.cc:288] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:19.905265 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 66c46bf28891458687ba0f3a17d0f733 (DEFAULT_TABLE table=default.Table_2e040bae1e56467b95bed23b37375389 [id=a3d530c0e8a048c9b0596045b191cfcf]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:19.905545 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1d432a59bdd344ddaaf710de73b543e6 consensus state has no leader
I20240507 08:12:19.907330 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5f9846e0eab74388af231e9942a6881a
W20240507 08:12:19.910686 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d9bae1b289994a2c8ea89be89ead6a7b consensus state has no leader
I20240507 08:12:19.910777 5685 tablet_service.cc:1467] Processing CreateTablet for tablet b0aaf4f1ff1841d6bd3eb9df30c39967 (DEFAULT_TABLE table=default.Table_896f00e31db14579b52336dddabe10ec [id=40c6022946364cc8968da1751275c8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.911274 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0aaf4f1ff1841d6bd3eb9df30c39967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.914232 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 66c46bf28891458687ba0f3a17d0f733 (DEFAULT_TABLE table=default.Table_2e040bae1e56467b95bed23b37375389 [id=a3d530c0e8a048c9b0596045b191cfcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.914505 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9846e0eab74388af231e9942a6881a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.914641 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c46bf28891458687ba0f3a17d0f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.917827 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9a64741db3c74294a4f2a1cde3d2b2e9 (DEFAULT_TABLE table=default.Table_0f1645256ab34e18a40eb96a15cac1de [id=476d0956eb3f463e82e5189d4f293e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.918090 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7] alter complete (version 1)
I20240507 08:12:19.918210 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a64741db3c74294a4f2a1cde3d2b2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.923147 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 6def4bf6a7554a4090a39b1823c4097c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.923605 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.924067 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f0b399a2d074f4ca3626ff17bd5518e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:07 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.924443 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 43225934cef04109b3dbc057b230f032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.924825 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet eec3704b135740f1a906d60f3ea87ffe with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.925030 5841 ts_tablet_manager.cc:1875] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.925218 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 156bf04b57724134a45fa9fc477596c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.925655 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a94c97ecc5d47abb190f3833fd317ca with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:19.926050 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c7f072f55a94cca938aa063637af9a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:19.929513 5256 catalog_manager.cc:5676] Failed to send AlterTable RPC for tablet 66247e08dfdf47f69ba24b074c8080f0 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 66247e08dfdf47f69ba24b074c8080f0 consensus state has no leader
I20240507 08:12:19.932945 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:19.933127 5447 raft_consensus.cc:3037] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:19.933436 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 9a64741db3c74294a4f2a1cde3d2b2e9 (DEFAULT_TABLE table=default.Table_0f1645256ab34e18a40eb96a15cac1de [id=476d0956eb3f463e82e5189d4f293e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.933688 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9846e0eab74388af231e9942a6881a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.950270 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 7b13eeb171b743b8b831d214d400ee7b (DEFAULT_TABLE table=default.Table_6362b6a3832a42099cbf4fc493c068fe [id=c097b927dfea4c698592f003be792282]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.950975 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b13eeb171b743b8b831d214d400ee7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.956035 5767 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:19.957767 6513 log.cc:1196] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8a015bf84a454d49b6497c16ed15c84a
I20240507 08:12:19.958344 6513 ts_tablet_manager.cc:1909] T 8a015bf84a454d49b6497c16ed15c84a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:19.958812 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1209) took 4192 ms. Trace:
I20240507 08:12:19.959018 6513 rpcz_store.cc:276] 0507 08:12:15.766568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.766643 (+ 75us) service_pool.cc:225] Handling call
0507 08:12:19.786766 (+4020123us) ts_tablet_manager.cc:1120] Deleting tablet 8a015bf84a454d49b6497c16ed15c84a
0507 08:12:19.786775 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.800268 (+ 13493us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.857181 (+ 56913us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.958753 (+101572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4020005}
I20240507 08:12:19.959618 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8a015bf84a454d49b6497c16ed15c84a (table default.Table_499cbcb48a784dcab2336eece710d4b1 [id=2516d64740a947508d9375e6c886dd59]) successfully deleted
I20240507 08:12:19.961258 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet a5976eca006448e0b9f2573acd738bb1 with delete_type TABLET_DATA_DELETED (replaced by tablet 5481ddbcd31449c4b44afb87a97138bd) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.902429 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 9a64741db3c74294a4f2a1cde3d2b2e9 (DEFAULT_TABLE table=default.Table_0f1645256ab34e18a40eb96a15cac1de [id=476d0956eb3f463e82e5189d4f293e3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.903856 5559 tablet_service.cc:1467] Processing CreateTablet for tablet caa86eebf05e42ac82af50878be87b60 (DEFAULT_TABLE table=default.Table_a14accce1d384c4b8f7772785c405082 [id=82a796de279d47368d198683af7e4bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.961493 5422 tablet_service.cc:1514] Processing DeleteTablet for tablet 43225934cef04109b3dbc057b230f032 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.961272 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 156bf04b57724134a45fa9fc477596c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.961938 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a64741db3c74294a4f2a1cde3d2b2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.904340 5552 tablet_service.cc:1467] Processing CreateTablet for tablet b0aaf4f1ff1841d6bd3eb9df30c39967 (DEFAULT_TABLE table=default.Table_896f00e31db14579b52336dddabe10ec [id=40c6022946364cc8968da1751275c8b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.963029 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0aaf4f1ff1841d6bd3eb9df30c39967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.950603 7378 raft_consensus.cc:482] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:19.963464 7378 raft_consensus.cc:504] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.964563 5420 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.914939 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "449d6c8270184ce2918cee8ce731f788" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:19.931175 5562 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9846e0eab74388af231e9942a6881a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:19 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.965752 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa86eebf05e42ac82af50878be87b60. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:19.966450 5654 leader_election.cc:341] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:19.931567 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 85dc58133d2b4f5496b4e570ae6494c0 with delete_type TABLET_DATA_DELETED (replaced by tablet 5c34a8ee0ca24e9490f3ba409b62a009) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:19.954092 5560 tablet_service.cc:1467] Processing CreateTablet for tablet 66c46bf28891458687ba0f3a17d0f733 (DEFAULT_TABLE table=default.Table_2e040bae1e56467b95bed23b37375389 [id=a3d530c0e8a048c9b0596045b191cfcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.960942 5839 tablet_bootstrap.cc:654] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.969775 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c46bf28891458687ba0f3a17d0f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:19.973794 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1215) took 2571 ms. Trace:
I20240507 08:12:19.973888 7079 rpcz_store.cc:276] 0507 08:12:17.402195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.402244 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:19.813115 (+2410871us) ts_tablet_manager.cc:1120] Deleting tablet 34adee00b529424b9f274c61ce73b0bc
0507 08:12:19.813123 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.829864 (+ 16741us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.851595 (+ 21731us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.973775 (+122180us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2410683}
I20240507 08:12:19.974242 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1214) took 2572 ms. Trace:
I20240507 08:12:19.974261 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 34adee00b529424b9f274c61ce73b0bc (table default.Table_548b730d6e06428c9f74b3cff2110d7d [id=3350995d44de43ee814fcf2ae9ca08c9]) successfully deleted
I20240507 08:12:19.974314 7079 rpcz_store.cc:276] 0507 08:12:17.402079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.402874 (+ 795us) service_pool.cc:225] Handling call
0507 08:12:19.974137 (+2571263us) ts_tablet_manager.cc:1120] Deleting tablet 71fcb6a8839840838bedb777ff1cbbcc
0507 08:12:19.974145 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.974227 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2571106}
I20240507 08:12:19.974397 5410 tablet_service.cc:1467] Processing CreateTablet for tablet caa86eebf05e42ac82af50878be87b60 (DEFAULT_TABLE table=default.Table_a14accce1d384c4b8f7772785c405082 [id=82a796de279d47368d198683af7e4bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.974560 7079 tablet_replica.cc:330] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:19.974689 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 71fcb6a8839840838bedb777ff1cbbcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71fcb6a8839840838bedb777ff1cbbcc already in progress: creating tablet
I20240507 08:12:19.974736 7079 raft_consensus.cc:2229] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:19.974781 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet d17b6a2f4aa14f2a927fa9ad56a62a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:10 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:19.975288 7079 raft_consensus.cc:2258] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.975445 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3df807895f4648f18a0eb83bb6307de0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.975812 7079 ts_tablet_manager.cc:1875] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:19.976058 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3df807895f4648f18a0eb83bb6307de0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.977301 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_f980c1cb998f49dea75f044a4201d5c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:19.977754 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f980c1cb998f49dea75f044a4201d5c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:19.979207 7649 consensus_queue.cc:1035] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:19.980226 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_709cc5845fc74e038578ef026cbb7567" } new_table_name: "default.Table_c7e3733ff21f4f19b964fae8b1150d8d" modify_external_catalogs: true
I20240507 08:12:19.981195 7079 ts_tablet_manager.cc:1888] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:19.981662 6513 tablet_replica.cc:330] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:19.981807 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5b56cd2bad1d4b3db77c63f023cd204c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:19.981949 6513 raft_consensus.cc:2229] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:19.982204 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b56cd2bad1d4b3db77c63f023cd204c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:19.982259 5654 consensus_peers.cc:485] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:19.982291 6513 raft_consensus.cc:2258] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:19.982826 6513 ts_tablet_manager.cc:1875] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:19.983232 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_fa93c4b1ea54406bb9beafa4a5d9bde6" } new_table_name: "default.Table_919a1a70d92647909eb6812decff79b6" modify_external_catalogs: true
I20240507 08:12:19.985558 5840 tablet_bootstrap.cc:492] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:19.985823 5840 ts_tablet_manager.cc:1367] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.183s user 0.026s sys 0.037s
I20240507 08:12:19.986694 5840 raft_consensus.cc:348] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.986974 5840 raft_consensus.cc:374] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:19.987067 5840 raft_consensus.cc:729] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:19.987299 5840 consensus_queue.cc:260] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:19.987896 5840 ts_tablet_manager.cc:1398] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:19.988950 5840 tablet_bootstrap.cc:492] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:19.990473 5840 tablet_bootstrap.cc:654] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:19.993119 5693 tablet_service.cc:1467] Processing CreateTablet for tablet caa86eebf05e42ac82af50878be87b60 (DEFAULT_TABLE table=default.Table_a14accce1d384c4b8f7772785c405082 [id=82a796de279d47368d198683af7e4bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.993665 5687 tablet_service.cc:1467] Processing CreateTablet for tablet a2acff4f495b4f45bcc477c7f245ffff (DEFAULT_TABLE table=default.Table_ebd7060e386f4e7cae3ae4ec34a2fa30 [id=54e9eff4f0f24d03b7643f3cb61c1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:19.994277 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 66247e08dfdf47f69ba24b074c8080f0 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]). Expected version 2 got 0
I20240507 08:12:19.996358 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 9a46fae5daa84e83882983b36d733d90 (DEFAULT_TABLE table=default.Table_30b57b7f7f384fda8a21c5bbd95dcdd8 [id=471d1bd257424d9992b0082ca1999f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.000494 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa86eebf05e42ac82af50878be87b60. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:20.002295 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4b78ec5ea7c43149c89be8dd2df0e45 consensus state has no leader
I20240507 08:12:20.002439 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2acff4f495b4f45bcc477c7f245ffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.004532 5263 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 66247e08dfdf47f69ba24b074c8080f0 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]). Expected version 2 got 0
I20240507 08:12:20.011557 5689 tablet_service.cc:1467] Processing CreateTablet for tablet f026605e491549cd97c0625476c8a974 (DEFAULT_TABLE table=default.Table_bec6672c91654fe68356ce93a8ea725a [id=0593b1bc47c04e41878d52cf5c60cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.013060 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 950b7ea014b14037b0ed6babb5963db5 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.015066 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a46fae5daa84e83882983b36d733d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.015518 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f1ba0da75f11411ab9babc78e508f6e8" } new_table_name: "default.Table_741fc840be7e49f5abe5ad28a15cec03" modify_external_catalogs: true
I20240507 08:12:20.017194 7649 raft_consensus.cc:482] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.017351 7649 raft_consensus.cc:504] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.017932 7649 leader_election.cc:288] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.018800 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_6e5b4736014d4aba946935a8e9530f78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.019326 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e5b4736014d4aba946935a8e9530f78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.019600 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_091c1a53ece34eb3bbc8265fa45f9088" } modify_external_catalogs: true
I20240507 08:12:20.020774 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d384a5d7c29a4df492a4cfaae5d18044" } modify_external_catalogs: true
I20240507 08:12:20.022830 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_45f5c047857b4e9fbcdc45084bdf09a4" } modify_external_catalogs: true
I20240507 08:12:20.024431 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0aaf4f1ff1841d6bd3eb9df30c39967. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.024906 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66c46bf28891458687ba0f3a17d0f733. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.025071 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a64741db3c74294a4f2a1cde3d2b2e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.025866 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet caa86eebf05e42ac82af50878be87b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.026988 7378 leader_election.cc:288] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.027463 5419 tablet_service.cc:1467] Processing CreateTablet for tablet a2acff4f495b4f45bcc477c7f245ffff (DEFAULT_TABLE table=default.Table_ebd7060e386f4e7cae3ae4ec34a2fa30 [id=54e9eff4f0f24d03b7643f3cb61c1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.028522 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43225934cef04109b3dbc057b230f032" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.028719 5718 raft_consensus.cc:2454] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.029117 5416 tablet_service.cc:1467] Processing CreateTablet for tablet 9a46fae5daa84e83882983b36d733d90 (DEFAULT_TABLE table=default.Table_30b57b7f7f384fda8a21c5bbd95dcdd8 [id=471d1bd257424d9992b0082ca1999f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.030226 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a46fae5daa84e83882983b36d733d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.030306 5377 leader_election.cc:302] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.030697 7378 raft_consensus.cc:2781] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.030602 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_412d04dd4e35482782b0d9ed337f3251"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.030761 7378 raft_consensus.cc:482] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.030817 7378 raft_consensus.cc:3037] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:20.031116 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_412d04dd4e35482782b0d9ed337f3251 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.035874 7079 log.cc:1196] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/565ac6091f5e48b1b024094d5560e275
I20240507 08:12:20.037410 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2acff4f495b4f45bcc477c7f245ffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.041793 5408 tablet_service.cc:1467] Processing CreateTablet for tablet f026605e491549cd97c0625476c8a974 (DEFAULT_TABLE table=default.Table_bec6672c91654fe68356ce93a8ea725a [id=0593b1bc47c04e41878d52cf5c60cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.042132 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f026605e491549cd97c0625476c8a974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.043238 7079 ts_tablet_manager.cc:1909] T 565ac6091f5e48b1b024094d5560e275 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
W20240507 08:12:20.046489 5654 consensus_peers.cc:485] T bcefbe3a60f84cadb5fd3e69c6efdd02 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 6: this message will repeat every 5th retry.
I20240507 08:12:20.048600 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9fcb6776bc164f07ba3ad8154d10bf36" } modify_external_catalogs: true
I20240507 08:12:20.048871 7378 raft_consensus.cc:504] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.049873 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43225934cef04109b3dbc057b230f032" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.050387 5378 leader_election.cc:341] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.050527 7378 leader_election.cc:288] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.050299 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1216) took 2583 ms. Trace:
I20240507 08:12:20.053473 5841 ts_tablet_manager.cc:1888] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:20.053751 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43225934cef04109b3dbc057b230f032" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.053973 5718 raft_consensus.cc:3037] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.056619 5718 raft_consensus.cc:2454] T 43225934cef04109b3dbc057b230f032 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.052368 7079 rpcz_store.cc:276] 0507 08:12:17.466987 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.467044 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:19.974522 (+2507478us) ts_tablet_manager.cc:1120] Deleting tablet 565ac6091f5e48b1b024094d5560e275
0507 08:12:19.974529 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:19.978596 (+ 4067us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:19.981124 (+ 2528us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.050279 (+ 69155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507304}
I20240507 08:12:19.985996 5559 tablet_service.cc:1467] Processing CreateTablet for tablet a2acff4f495b4f45bcc477c7f245ffff (DEFAULT_TABLE table=default.Table_ebd7060e386f4e7cae3ae4ec34a2fa30 [id=54e9eff4f0f24d03b7643f3cb61c1ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.057053 5377 leader_election.cc:302] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.057298 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2acff4f495b4f45bcc477c7f245ffff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.057664 7378 raft_consensus.cc:2781] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.057786 7378 raft_consensus.cc:686] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.058029 7378 consensus_queue.cc:237] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.058338 5841 log.cc:1196] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/08aac6e88b7c41d8a59d1e12e7f6c14e
I20240507 08:12:20.058696 5841 ts_tablet_manager.cc:1909] T 08aac6e88b7c41d8a59d1e12e7f6c14e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.058984 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1224) took 3495 ms. Trace:
I20240507 08:12:20.059053 5841 rpcz_store.cc:276] 0507 08:12:16.563924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.579864 (+ 15940us) service_pool.cc:225] Handling call
0507 08:12:19.886937 (+3307073us) ts_tablet_manager.cc:1120] Deleting tablet 08aac6e88b7c41d8a59d1e12e7f6c14e
0507 08:12:19.886946 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.033531 (+146585us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.053387 (+ 19856us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.058968 (+ 5581us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3306883}
I20240507 08:12:20.059445 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1227) took 3405 ms. Trace:
I20240507 08:12:20.059509 5841 rpcz_store.cc:276] 0507 08:12:16.653769 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.653821 (+ 52us) service_pool.cc:225] Handling call
0507 08:12:20.059323 (+3405502us) ts_tablet_manager.cc:1120] Deleting tablet 9669c1169b5244218383f7aa7fbc3b85
0507 08:12:20.059332 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.059429 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3405295}
I20240507 08:12:20.059799 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1226) took 3406 ms. Trace:
I20240507 08:12:20.059862 5841 rpcz_store.cc:276] 0507 08:12:16.653637 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.653826 (+ 189us) service_pool.cc:225] Handling call
0507 08:12:20.059709 (+3405883us) ts_tablet_manager.cc:1120] Deleting tablet 73f82413951c48a89e7db8fe9e50c749
0507 08:12:20.059716 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.059787 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3405618}
I20240507 08:12:20.060142 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1228) took 3404 ms. Trace:
I20240507 08:12:20.059177 6513 ts_tablet_manager.cc:1888] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.060207 5841 rpcz_store.cc:276] 0507 08:12:16.655572 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.655619 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:20.060055 (+3404436us) ts_tablet_manager.cc:1120] Deleting tablet 994f50ea19a2445288663b7d7b7ad291
0507 08:12:20.060063 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.060125 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3404272}
I20240507 08:12:20.060247 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 565ac6091f5e48b1b024094d5560e275 (table default.Table_457ae0dd8b974005872846c9fe575b10 [id=796006aacf9741f8a4dc1d5916887655]) successfully deleted
I20240507 08:12:20.060500 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1233) took 3327 ms. Trace:
I20240507 08:12:20.060566 5841 rpcz_store.cc:276] 0507 08:12:16.732931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.732991 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:20.060407 (+3327416us) ts_tablet_manager.cc:1120] Deleting tablet 17c6f752a0d4485e9606a5570a0d6c96
0507 08:12:20.060418 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.060486 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202679}
I20240507 08:12:20.061450 6513 log.cc:1196] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/3f01e3311a8b4553a31914922fb08d13
I20240507 08:12:20.062812 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66247e08dfdf47f69ba24b074c8080f0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.060804 5843 tablet_bootstrap.cc:654] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.063010 5445 raft_consensus.cc:2454] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.061780 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1217) took 2594 ms. Trace:
I20240507 08:12:19.998317 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 9a46fae5daa84e83882983b36d733d90 (DEFAULT_TABLE table=default.Table_30b57b7f7f384fda8a21c5bbd95dcdd8 [id=471d1bd257424d9992b0082ca1999f93]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.064195 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9a46fae5daa84e83882983b36d733d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.064543 5841 tablet_replica.cc:330] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.065150 5841 raft_consensus.cc:2229] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.065860 5841 raft_consensus.cc:2258] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.063686 7079 rpcz_store.cc:276] 0507 08:12:17.467510 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.467556 (+ 46us) service_pool.cc:225] Handling call
0507 08:12:20.059529 (+2591973us) ts_tablet_manager.cc:1120] Deleting tablet 3ef4375925474e789a39a2c28fa6adfe
0507 08:12:20.059544 (+ 15us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.061760 (+ 2216us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2590913}
I20240507 08:12:20.068373 5841 ts_tablet_manager.cc:1875] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:20.068981 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 3ef4375925474e789a39a2c28fa6adfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef4375925474e789a39a2c28fa6adfe already in progress: creating tablet
I20240507 08:12:20.003429 5582 raft_consensus.cc:3037] T 8bb62a72ed2248f6886823b9c64ff4c6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.070314 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f026605e491549cd97c0625476c8a974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.070806 6513 ts_tablet_manager.cc:1909] T 3f01e3311a8b4553a31914922fb08d13 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.010432 5548 tablet_service.cc:1467] Processing CreateTablet for tablet f026605e491549cd97c0625476c8a974 (DEFAULT_TABLE table=default.Table_bec6672c91654fe68356ce93a8ea725a [id=0593b1bc47c04e41878d52cf5c60cce9]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.012274 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 950b7ea014b14037b0ed6babb5963db5 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.018522 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66247e08dfdf47f69ba24b074c8080f0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.072455 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f026605e491549cd97c0625476c8a974. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.072314 7079 tablet_replica.cc:330] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:20.073781 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950b7ea014b14037b0ed6babb5963db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.018512 5839 tablet_bootstrap.cc:492] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.075194 5839 ts_tablet_manager.cc:1367] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.211s user 0.007s sys 0.001s
I20240507 08:12:20.075937 5839 raft_consensus.cc:348] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.076121 5839 raft_consensus.cc:374] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.076165 5839 raft_consensus.cc:729] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.076349 5839 consensus_queue.cc:260] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.024400 7602 raft_consensus.cc:482] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.076753 7602 raft_consensus.cc:504] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.077327 7602 leader_election.cc:288] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.077672 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950b7ea014b14037b0ed6babb5963db5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:20.078570 5520 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 33875331045146a9a02559a0b37d5c6f
I20240507 08:12:20.077919 7079 raft_consensus.cc:2229] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.077855 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1204) took 4311 ms. Trace:
I20240507 08:12:20.028016 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43225934cef04109b3dbc057b230f032" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.080956 5654 leader_election.cc:341] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.082243 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]) successfully deleted
I20240507 08:12:20.081255 7079 raft_consensus.cc:2258] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:20.082494 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9669c1169b5244218383f7aa7fbc3b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9669c1169b5244218383f7aa7fbc3b85 already in progress: creating tablet
W20240507 08:12:20.082685 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 73f82413951c48a89e7db8fe9e50c749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f82413951c48a89e7db8fe9e50c749 already in progress: creating tablet
W20240507 08:12:20.082846 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 994f50ea19a2445288663b7d7b7ad291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994f50ea19a2445288663b7d7b7ad291 already in progress: creating tablet
W20240507 08:12:20.082998 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 17c6f752a0d4485e9606a5570a0d6c96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17c6f752a0d4485e9606a5570a0d6c96 already in progress: creating tablet
I20240507 08:12:20.079896 6513 rpcz_store.cc:276] 0507 08:12:15.765990 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.766731 (+ 741us) service_pool.cc:225] Handling call
0507 08:12:19.981595 (+4214864us) ts_tablet_manager.cc:1120] Deleting tablet 3f01e3311a8b4553a31914922fb08d13
0507 08:12:19.981604 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.000253 (+ 18649us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.056552 (+ 56299us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.077112 (+ 20560us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4214679}
I20240507 08:12:20.085791 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 3f01e3311a8b4553a31914922fb08d13 (table default.Table_21a782d34aee4be78034577651967f0b [id=83ce25b981d143f294b593fa17ea57f0]) successfully deleted
W20240507 08:12:20.080526 5378 leader_election.cc:341] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.082337 7499 raft_consensus.cc:482] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.086431 5656 leader_election.cc:302] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.087116 7579 raft_consensus.cc:2781] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.087229 7579 raft_consensus.cc:482] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.087322 7579 raft_consensus.cc:3037] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.090844 7079 ts_tablet_manager.cc:1875] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.091740 7579 raft_consensus.cc:504] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.092617 7579 leader_election.cc:288] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.092751 6513 tablet_replica.cc:330] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.092957 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66247e08dfdf47f69ba24b074c8080f0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.093286 6513 raft_consensus.cc:2229] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:20.093771 5654 leader_election.cc:341] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.094300 5517 leader_election.cc:341] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 33875331045146a9a02559a0b37d5c6f
I20240507 08:12:20.094432 5517 leader_election.cc:302] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.094820 7602 raft_consensus.cc:2726] T 33875331045146a9a02559a0b37d5c6f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:20.100428 5656 consensus_peers.cc:485] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43644b4405f44bfd9a1afcaedbd68db5. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.105638 6513 raft_consensus.cc:2258] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.107285 6513 ts_tablet_manager.cc:1875] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.107336 5262 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_36650d696e794757a41588fcf93d8b69" } new_table_name: "default.Table_ebd20a6623a141d3a8dad4e150ef3441" modify_external_catalogs: true
I20240507 08:12:20.111338 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2db26a18c77448aba96e29400de518e" } modify_external_catalogs: true
I20240507 08:12:20.112124 5447 raft_consensus.cc:2454] T 449d6c8270184ce2918cee8ce731f788 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.086426 7499 raft_consensus.cc:504] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.114838 5656 leader_election.cc:302] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.115614 5840 tablet_bootstrap.cc:492] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.115653 7579 raft_consensus.cc:2781] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.115990 7579 raft_consensus.cc:686] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.116099 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 950b7ea014b14037b0ed6babb5963db5 (DEFAULT_TABLE table=default.Table_1f68c8dc6aa54480bac7fcc19cd692e7 [id=7ecb10d36ca944649cd114ed7c4396f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:20.116119 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.116007 5840 ts_tablet_manager.cc:1367] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.127s user 0.011s sys 0.004s
I20240507 08:12:20.117193 7579 consensus_queue.cc:237] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.116999 5378 leader_election.cc:341] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.125710 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950b7ea014b14037b0ed6babb5963db5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:20.132040 5840 raft_consensus.cc:348] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.136157 5840 raft_consensus.cc:374] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.100066 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66247e08dfdf47f69ba24b074c8080f0" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.138273 5442 raft_consensus.cc:3037] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.130831 7499 leader_election.cc:288] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.138353 5840 raft_consensus.cc:729] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.143080 7649 raft_consensus.cc:482] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.143251 7649 raft_consensus.cc:504] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.144006 7649 leader_election.cc:288] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.145511 5442 raft_consensus.cc:2454] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.148419 5656 leader_election.cc:302] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.149089 7649 raft_consensus.cc:2781] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.149223 7649 raft_consensus.cc:686] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.149286 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.149482 7649 consensus_queue.cc:237] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.149840 5442 raft_consensus.cc:2454] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.150754 5656 leader_election.cc:302] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.151235 6675 raft_consensus.cc:2781] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.151326 6675 raft_consensus.cc:482] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.151427 6675 raft_consensus.cc:3037] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.153263 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_45f5c047857b4e9fbcdc45084bdf09a4" } modify_external_catalogs: true
I20240507 08:12:20.154572 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d384a5d7c29a4df492a4cfaae5d18044" } modify_external_catalogs: true
I20240507 08:12:20.154656 6675 raft_consensus.cc:504] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.155282 6675 leader_election.cc:288] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.156035 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_091c1a53ece34eb3bbc8265fa45f9088" } modify_external_catalogs: true
I20240507 08:12:20.147954 5840 consensus_queue.cc:260] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.161523 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.161718 5442 raft_consensus.cc:3037] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.164752 5442 raft_consensus.cc:2454] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.165375 5656 leader_election.cc:302] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.165787 6675 raft_consensus.cc:2781] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.166131 6675 raft_consensus.cc:686] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.166009 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8f5d3a9c55b34fa492ef33de0b68ae95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.166437 6675 consensus_queue.cc:237] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.166656 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f5d3a9c55b34fa492ef33de0b68ae95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.172619 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5370d022aa904900a28842b887b13187
I20240507 08:12:20.173375 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_8a6005ba074147a5a478af48005277b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.173890 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a6005ba074147a5a478af48005277b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.175629 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 5370d022aa904900a28842b887b13187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.176832 7358 raft_consensus.cc:482] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.176985 7358 raft_consensus.cc:504] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.177561 7358 leader_election.cc:288] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.177884 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c42f6e958dd94e3cbecd5c4bd6a6d829
I20240507 08:12:20.178454 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0f89d6746a9d46a7a74a488cdffea6a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.178706 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:20.178891 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f89d6746a9d46a7a74a488cdffea6a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.180091 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 3f30af566be44760828e1672d83fc903
I20240507 08:12:20.180697 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1aa1e107154242ad9d480465921434b1" } new_table_name: "default.Table_ba2266c5bc18454c82c48b6490d39b6f" modify_external_catalogs: true
I20240507 08:12:20.145035 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.183499 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d2db26a18c77448aba96e29400de518e" } modify_external_catalogs: true
I20240507 08:12:20.183779 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f30af566be44760828e1672d83fc903 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.184203 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee9e434083f4fa6859cfe2b7cca16e5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.184423 5409 tablet_service.cc:1514] Processing DeleteTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:20.184742 5656 leader_election.cc:341] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: Tablet not RUNNING: STOPPED
I20240507 08:12:20.185068 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet a605be6967a041388dfe81053fbb602e (table default.Table_13bfcb3083e54a37b5fad286ae0e5b26 [id=b92037dbac744552b58099b7fdc424b2]). Expected version 1 got 0
I20240507 08:12:20.187784 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f30af566be44760828e1672d83fc903 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.188195 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 5370d022aa904900a28842b887b13187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.190477 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5db778c4573e453e9175054014cf7070"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.191016 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5db778c4573e453e9175054014cf7070 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.192030 7358 raft_consensus.cc:482] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.192170 7358 raft_consensus.cc:504] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.192740 7358 leader_election.cc:288] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:20.193331 5262 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 437fafdae750487ca4655d6a97c230d1 consensus state has no leader
I20240507 08:12:20.193333 5841 ts_tablet_manager.cc:1888] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.193459 5841 log.cc:1196] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/bee9e434083f4fa6859cfe2b7cca16e5
I20240507 08:12:20.193821 5841 ts_tablet_manager.cc:1909] T bee9e434083f4fa6859cfe2b7cca16e5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.193732 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9b144d4b0c8f47f5bb7a5d8bfba7bf59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.194124 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1234) took 3411 ms. Trace:
I20240507 08:12:20.194646 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9fcb6776bc164f07ba3ad8154d10bf36" } modify_external_catalogs: true
W20240507 08:12:20.197757 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b144d4b0c8f47f5bb7a5d8bfba7bf59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.198596 5840 ts_tablet_manager.cc:1398] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.082s user 0.002s sys 0.002s
W20240507 08:12:20.198619 5768 tablet.cc:2369] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:12:20.198922 5840 tablet_bootstrap.cc:492] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.200632 7358 raft_consensus.cc:482] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.200798 7358 raft_consensus.cc:504] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.201429 7358 leader_election.cc:288] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.202150 5840 tablet_bootstrap.cc:654] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.203639 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_43d14c4de78b401692f0085440466c11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.204129 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_43d14c4de78b401692f0085440466c11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.205674 5841 rpcz_store.cc:276] 0507 08:12:16.782294 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.782351 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:20.062927 (+3280576us) ts_tablet_manager.cc:1120] Deleting tablet bee9e434083f4fa6859cfe2b7cca16e5
0507 08:12:20.062938 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.112391 (+ 49453us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.193237 (+ 80846us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.194082 (+ 845us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3205123}
I20240507 08:12:20.206156 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet bee9e434083f4fa6859cfe2b7cca16e5 (table default.Table_615d90859dd74c9aa813e233b63dd68e [id=0652c43a334e4f04a34ff59ca43e545a]) successfully deleted
I20240507 08:12:20.206445 5841 tablet_replica.cc:330] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.206643 5841 raft_consensus.cc:2229] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.206954 5841 raft_consensus.cc:2258] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.207469 5841 ts_tablet_manager.cc:1875] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.211217 7358 raft_consensus.cc:482] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.211369 7358 raft_consensus.cc:504] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.211927 7358 leader_election.cc:288] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:12:20.212306 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 140b4601f1a043329c4de612a3ae244c consensus state has no leader
I20240507 08:12:20.217790 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 9354ceaf6f6d4f529d9dbe7b78360b82
I20240507 08:12:20.218538 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.220185 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.222533 7079 ts_tablet_manager.cc:1888] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.223650 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_562f2b2c8b3e4c0eafa7e7cd93389685"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.224162 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_562f2b2c8b3e4c0eafa7e7cd93389685 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.224488 5260 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_e174ec7e722344baa23361800c5780d4" } new_table_name: "default.Table_0349bf7e0f9e4be394993e0f579a50b9" modify_external_catalogs: true
I20240507 08:12:20.193568 5843 tablet_bootstrap.cc:492] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.226343 5843 ts_tablet_manager.cc:1367] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.330s user 0.000s sys 0.010s
I20240507 08:12:20.227006 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 401402afc04f4a9082c02ca5bf39d6a1
I20240507 08:12:20.227074 5843 raft_consensus.cc:348] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.227308 5843 raft_consensus.cc:374] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.227406 5843 raft_consensus.cc:729] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.227640 5843 consensus_queue.cc:260] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.228250 5843 ts_tablet_manager.cc:1398] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240507 08:12:20.230181 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.230882 5841 ts_tablet_manager.cc:1888] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.231025 5841 log.cc:1196] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/242ea35c61404a91b1211c5ad9207728
I20240507 08:12:20.231417 5841 ts_tablet_manager.cc:1909] T 242ea35c61404a91b1211c5ad9207728 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.232753 5840 tablet_bootstrap.cc:492] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.232951 5840 ts_tablet_manager.cc:1367] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20240507 08:12:20.233924 5840 raft_consensus.cc:348] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.234500 5840 raft_consensus.cc:374] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.234599 5840 raft_consensus.cc:729] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.234890 5840 consensus_queue.cc:260] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.236092 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f2867d292c9d45a3b77c426279d29c32" } new_table_name: "default.Table_26149001a1a44a28865ac44532375eca" modify_external_catalogs: true
I20240507 08:12:20.230046 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.245339 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b6a2f4aa14f2a927fa9ad56a62a5a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.238744 5584 raft_consensus.cc:3037] T ba28c36f91e141f4a1372e7544b44281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.239619 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bee9e434083f4fa6859cfe2b7cca16e5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.237288 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a94c97ecc5d47abb190f3833fd317ca" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.239969 5839 ts_tablet_manager.cc:1398] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.165s user 0.007s sys 0.000s
I20240507 08:12:20.246484 5839 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.245102 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.249550 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.222697 7079 log.cc:1196] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/28e5a380f2784e3a806afcbb5cd2519a
I20240507 08:12:20.258074 7079 ts_tablet_manager.cc:1909] T 28e5a380f2784e3a806afcbb5cd2519a P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:20.258587 7725 raft_consensus.cc:482] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.258759 7725 raft_consensus.cc:504] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.259630 7725 leader_election.cc:288] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.266252 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1219) took 2711 ms. Trace:
I20240507 08:12:20.266407 7079 rpcz_store.cc:276] 0507 08:12:17.555117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.565637 (+ 10520us) service_pool.cc:225] Handling call
0507 08:12:20.071659 (+2506022us) ts_tablet_manager.cc:1120] Deleting tablet 28e5a380f2784e3a806afcbb5cd2519a
0507 08:12:20.071673 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.190583 (+118910us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.222436 (+ 31853us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.266232 (+ 43796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2476197}
I20240507 08:12:20.266891 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1220) took 2663 ms. Trace:
I20240507 08:12:20.267017 7079 rpcz_store.cc:276] 0507 08:12:17.603171 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.603523 (+ 352us) service_pool.cc:225] Handling call
0507 08:12:20.266777 (+2663254us) ts_tablet_manager.cc:1120] Deleting tablet c9a65a8e939f4b7d89d00720845ea1fa
0507 08:12:20.266787 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.266875 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2663039}
I20240507 08:12:20.267390 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1229) took 2609 ms. Trace:
I20240507 08:12:20.267354 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 28e5a380f2784e3a806afcbb5cd2519a (table default.Table_8a58f4e04c4147368d43d162c9613c9f [id=0e33fa8f8fff4e09916ca2783a4903e7]) successfully deleted
I20240507 08:12:20.267509 7079 rpcz_store.cc:276] 0507 08:12:17.658180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.658347 (+ 167us) service_pool.cc:225] Handling call
0507 08:12:20.267296 (+2608949us) ts_tablet_manager.cc:1120] Deleting tablet d8e03f8655714d309d5c8ae49c4b314b
0507 08:12:20.267304 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.267378 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2608779}
W20240507 08:12:20.267624 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c9a65a8e939f4b7d89d00720845ea1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a65a8e939f4b7d89d00720845ea1fa already in progress: creating tablet
I20240507 08:12:20.267889 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1230) took 2609 ms. Trace:
I20240507 08:12:20.267963 7079 rpcz_store.cc:276] 0507 08:12:17.658300 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.658552 (+ 252us) service_pool.cc:225] Handling call
0507 08:12:20.267796 (+2609244us) ts_tablet_manager.cc:1120] Deleting tablet fe721ba2f0ed4a3a887e7496e85d163c
0507 08:12:20.267808 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.267875 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2609084}
W20240507 08:12:20.268227 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d8e03f8655714d309d5c8ae49c4b314b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e03f8655714d309d5c8ae49c4b314b already in progress: creating tablet
W20240507 08:12:20.270460 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68981cf4d9e74d41b55c411b4c678dab consensus state has no leader
I20240507 08:12:20.271279 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.271824 5378 leader_election.cc:341] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.275113 5439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.275286 5439 raft_consensus.cc:2454] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.275462 5447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b6a2f4aa14f2a927fa9ad56a62a5a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.275605 5447 raft_consensus.cc:2454] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.278522 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.278744 5442 raft_consensus.cc:2454] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.282738 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1235) took 3484 ms. Trace:
I20240507 08:12:20.282833 5843 tablet_bootstrap.cc:492] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.274833 5264 catalog_manager.cc:5484] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:20.283035 5264 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 449d6c8270184ce2918cee8ce731f788 (table default.Table_2c4d993ae928415181f53571b6bf0815 [id=3764a23f5a7847c8b46385bdfaf51f20]). Expected version 1 got 0
I20240507 08:12:20.282836 5841 rpcz_store.cc:276] 0507 08:12:16.798793 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.798852 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:20.206388 (+3407536us) ts_tablet_manager.cc:1120] Deleting tablet 242ea35c61404a91b1211c5ad9207728
0507 08:12:20.206398 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.219299 (+ 12901us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.230778 (+ 11479us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.282721 (+ 51943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3337026}
I20240507 08:12:20.283407 5841 tablet_replica.cc:330] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.283586 5841 raft_consensus.cc:2229] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.283721 5841 raft_consensus.cc:2258] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.284116 5843 tablet_bootstrap.cc:654] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.284230 5841 ts_tablet_manager.cc:1875] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.251617 6675 raft_consensus.cc:482] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.252532 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5370d022aa904900a28842b887b13187 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.252705 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.252848 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f30af566be44760828e1672d83fc903 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.285321 6675 raft_consensus.cc:504] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.285933 6675 leader_election.cc:288] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.286533 5840 ts_tablet_manager.cc:1398] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.053s user 0.000s sys 0.002s
I20240507 08:12:20.256567 7732 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:20.270445 6513 ts_tablet_manager.cc:1888] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.279471 7579 consensus_queue.cc:1035] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:20.287206 6513 log.cc:1196] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/4e5439844f384513b4719cfe4f4e5d9b
I20240507 08:12:20.287614 6513 ts_tablet_manager.cc:1909] T 4e5439844f384513b4719cfe4f4e5d9b P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.288352 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1208) took 4521 ms. Trace:
I20240507 08:12:20.288453 6513 rpcz_store.cc:276] 0507 08:12:15.766454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.767162 (+ 708us) service_pool.cc:225] Handling call
0507 08:12:20.092539 (+4325377us) ts_tablet_manager.cc:1120] Deleting tablet 4e5439844f384513b4719cfe4f4e5d9b
0507 08:12:20.092548 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.203752 (+111204us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.270294 (+ 66542us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.288297 (+ 18003us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4325180}
I20240507 08:12:20.288740 6513 tablet_replica.cc:330] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.288920 6513 raft_consensus.cc:2229] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.289213 6513 raft_consensus.cc:2258] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.279731 5656 leader_election.cc:302] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.289765 7579 raft_consensus.cc:2781] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.289851 7579 raft_consensus.cc:482] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.289973 7579 raft_consensus.cc:3037] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.291446 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 242ea35c61404a91b1211c5ad9207728 (table default.Table_5d4ba83817d04005be08381a5c9c15b3 [id=a79c79866e6547a2b0ea94eeaec8c98e]) successfully deleted
I20240507 08:12:20.292340 7579 raft_consensus.cc:504] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.292975 7579 leader_election.cc:288] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.294337 5656 leader_election.cc:302] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.294582 7579 raft_consensus.cc:2781] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.294670 7579 raft_consensus.cc:482] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.295804 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:20.295893 5654 leader_election.cc:341] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.296028 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 401402afc04f4a9082c02ca5bf39d6a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:20.296284 5654 leader_election.cc:341] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.296321 5841 ts_tablet_manager.cc:1888] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240507 08:12:20.296409 5654 leader_election.cc:341] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.296478 5654 leader_election.cc:302] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.296535 5841 log.cc:1196] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/167ee9cbc9174b82824c468e098472a6
W20240507 08:12:20.296772 5654 leader_election.cc:341] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.297011 5654 consensus_peers.cc:485] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e17502bb4ac64f6ca9c77a3b00ab9f4a. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.297039 5841 ts_tablet_manager.cc:1909] T 167ee9cbc9174b82824c468e098472a6 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:20.297119 5654 leader_election.cc:341] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.297323 5654 leader_election.cc:341] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.297492 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1236) took 3497 ms. Trace:
I20240507 08:12:20.297545 7649 raft_consensus.cc:2726] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.297734 5841 rpcz_store.cc:276] 0507 08:12:16.799624 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.799696 (+ 72us) service_pool.cc:225] Handling call
0507 08:12:20.283366 (+3483670us) ts_tablet_manager.cc:1120] Deleting tablet 167ee9cbc9174b82824c468e098472a6
0507 08:12:20.283375 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.292900 (+ 9525us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.296235 (+ 3335us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.297473 (+ 1238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3413933}
I20240507 08:12:20.298345 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 167ee9cbc9174b82824c468e098472a6 (table default.Table_3ad8e609d07646e49e01cc55ba8ee969 [id=c5ee2823c21745f380337f43009224f7]) successfully deleted
I20240507 08:12:20.298918 5841 tablet_replica.cc:330] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.299132 5841 raft_consensus.cc:2229] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.299456 5841 raft_consensus.cc:2258] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.300603 5256 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_e3bcf078654d42ed8331d033b29e482c [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 2 got 0
I20240507 08:12:20.301208 5841 ts_tablet_manager.cc:1875] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.301721 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5012e339983849aea31fac5699f84ccd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.302265 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5012e339983849aea31fac5699f84ccd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.302317 5656 leader_election.cc:302] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.302570 7656 raft_consensus.cc:2781] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.302644 7656 raft_consensus.cc:482] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.302716 7656 raft_consensus.cc:3037] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.302655 7736 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:20.305090 7656 raft_consensus.cc:504] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.305850 7656 leader_election.cc:288] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.306593 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.306828 5446 raft_consensus.cc:2454] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.309238 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.309466 5446 raft_consensus.cc:3037] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.310456 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.310621 5445 raft_consensus.cc:3037] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.310828 5839 tablet_bootstrap.cc:654] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.311144 5656 leader_election.cc:302] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.311865 7656 raft_consensus.cc:2781] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.311926 7656 raft_consensus.cc:482] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.311978 7656 raft_consensus.cc:3037] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.311942 5446 raft_consensus.cc:2454] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.313198 5445 raft_consensus.cc:2454] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.314383 7656 raft_consensus.cc:504] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.315107 7656 leader_election.cc:288] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.315990 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.316161 5445 raft_consensus.cc:3037] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.316816 5656 leader_election.cc:302] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.317567 6675 raft_consensus.cc:2781] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.317849 6675 raft_consensus.cc:686] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.318122 6675 consensus_queue.cc:237] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.318919 5445 raft_consensus.cc:2454] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.323691 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b31fdadee60e4f5e94e92c536e18774d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.324100 5656 leader_election.cc:302] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.324369 7656 raft_consensus.cc:2781] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.324518 7656 raft_consensus.cc:686] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.324757 7656 consensus_queue.cc:237] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.329175 5841 ts_tablet_manager.cc:1888] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.329231 5656 leader_election.cc:302] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.331725 7649 raft_consensus.cc:2781] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.331888 7649 raft_consensus.cc:686] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.332165 7649 consensus_queue.cc:237] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.333832 5841 log.cc:1196] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c534708f1abd4a33965eea92bdc0d0de
I20240507 08:12:20.334267 5841 ts_tablet_manager.cc:1909] T c534708f1abd4a33965eea92bdc0d0de P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.334611 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1237) took 3534 ms. Trace:
I20240507 08:12:20.334738 5841 rpcz_store.cc:276] 0507 08:12:16.800392 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.800445 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:20.298873 (+3498428us) ts_tablet_manager.cc:1120] Deleting tablet c534708f1abd4a33965eea92bdc0d0de
0507 08:12:20.298883 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.324069 (+ 25186us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.329099 (+ 5030us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.334594 (+ 5495us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3429399}
I20240507 08:12:20.335229 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c534708f1abd4a33965eea92bdc0d0de (table default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]) successfully deleted
I20240507 08:12:20.335664 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1241) took 3519 ms. Trace:
I20240507 08:12:20.335785 5841 rpcz_store.cc:276] 0507 08:12:16.816274 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.816331 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:20.335541 (+3519210us) ts_tablet_manager.cc:1120] Deleting tablet a68badb3b5644ed8b007b2b04bc4f6e1
0507 08:12:20.335551 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.335648 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3457377}
W20240507 08:12:20.336252 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a68badb3b5644ed8b007b2b04bc4f6e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68badb3b5644ed8b007b2b04bc4f6e1 already in progress: creating tablet
I20240507 08:12:20.336628 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1242) took 3455 ms. Trace:
I20240507 08:12:20.336772 5841 rpcz_store.cc:276] 0507 08:12:16.881182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.881245 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:20.336527 (+3455282us) ts_tablet_manager.cc:1120] Deleting tablet 219bad2b972c45a8930d6f47ca361858
0507 08:12:20.336535 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.336613 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3455087}
W20240507 08:12:20.337785 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 219bad2b972c45a8930d6f47ca361858 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219bad2b972c45a8930d6f47ca361858 already in progress: creating tablet
I20240507 08:12:20.338152 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1245) took 3437 ms. Trace:
I20240507 08:12:20.338268 5841 rpcz_store.cc:276] 0507 08:12:16.900583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.903594 (+ 3011us) service_pool.cc:225] Handling call
0507 08:12:20.338053 (+3434459us) ts_tablet_manager.cc:1120] Deleting tablet 3954a4e0d3104161aad427eb2781d55d
0507 08:12:20.338062 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.338137 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3434207}
W20240507 08:12:20.338680 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3954a4e0d3104161aad427eb2781d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3954a4e0d3104161aad427eb2781d55d already in progress: creating tablet
I20240507 08:12:20.338976 5841 tablet_replica.cc:330] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.339207 5841 raft_consensus.cc:2229] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:20.341236 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 43791.4us
I20240507 08:12:20.339577 5841 raft_consensus.cc:2258] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.343559 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:20.343596 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_e3bcf078654d42ed8331d033b29e482c [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 2 got 0
I20240507 08:12:20.344727 5841 ts_tablet_manager.cc:1875] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.356706 5841 ts_tablet_manager.cc:1888] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.356896 5841 log.cc:1196] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/7d89e4185229431f9a68f599da4aff34
I20240507 08:12:20.357318 5841 ts_tablet_manager.cc:1909] T 7d89e4185229431f9a68f599da4aff34 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.358947 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1248) took 3407 ms. Trace:
I20240507 08:12:20.359083 5841 rpcz_store.cc:276] 0507 08:12:16.951386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.951454 (+ 68us) service_pool.cc:225] Handling call
0507 08:12:20.338936 (+3387482us) ts_tablet_manager.cc:1120] Deleting tablet 7d89e4185229431f9a68f599da4aff34
0507 08:12:20.338945 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.350439 (+ 11494us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.356610 (+ 6171us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.358921 (+ 2311us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3387106}
I20240507 08:12:20.359534 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 7d89e4185229431f9a68f599da4aff34 (table default.Table_c290c4af761c45e885f1f4af9a48f4fc [id=c4b80c0e3c6847e6a03dc4cdf6989c67]) successfully deleted
I20240507 08:12:20.359863 5841 tablet_replica.cc:330] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.360090 5841 raft_consensus.cc:2229] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.360967 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1228) took 2702 ms. Trace:
W20240507 08:12:20.361054 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet fe721ba2f0ed4a3a887e7496e85d163c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe721ba2f0ed4a3a887e7496e85d163c already in progress: creating tablet
I20240507 08:12:20.361057 7079 rpcz_store.cc:276] 0507 08:12:17.658060 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.666500 (+ 8440us) service_pool.cc:225] Handling call
0507 08:12:20.360859 (+2694359us) ts_tablet_manager.cc:1120] Deleting tablet 4b78c08d758043c78d5bb8281b7f2d62
0507 08:12:20.360868 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.360950 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2694211}
I20240507 08:12:20.361203 5841 raft_consensus.cc:2258] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.361379 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1236) took 2669 ms. Trace:
I20240507 08:12:20.361519 7079 rpcz_store.cc:276] 0507 08:12:17.691955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.691998 (+ 43us) service_pool.cc:225] Handling call
0507 08:12:20.361285 (+2669287us) ts_tablet_manager.cc:1120] Deleting tablet cc3d1e9e7e53483fa75f7cd918295e50
0507 08:12:20.361293 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.361364 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2669047}
I20240507 08:12:20.361802 7079 tablet_replica.cc:330] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:20.361927 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4b78c08d758043c78d5bb8281b7f2d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b78c08d758043c78d5bb8281b7f2d62 already in progress: creating tablet
I20240507 08:12:20.362032 5841 ts_tablet_manager.cc:1875] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.362107 7079 raft_consensus.cc:2229] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:20.362224 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cc3d1e9e7e53483fa75f7cd918295e50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc3d1e9e7e53483fa75f7cd918295e50 already in progress: creating tablet
I20240507 08:12:20.362264 7079 raft_consensus.cc:2258] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.362726 7079 ts_tablet_manager.cc:1875] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.286839 5840 tablet_bootstrap.cc:492] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:20.366925 5260 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet de7cddb2c4f743f7ae02fd92061a418c consensus state has no leader
I20240507 08:12:20.367408 5840 tablet_bootstrap.cc:654] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:20.373674 5656 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 44454.6us
I20240507 08:12:20.381563 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.381827 5718 raft_consensus.cc:2454] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.384519 7725 raft_consensus.cc:482] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.384579 5839 tablet_bootstrap.cc:492] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.384699 5839 ts_tablet_manager.cc:1367] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.138s user 0.004s sys 0.000s
I20240507 08:12:20.384665 7725 raft_consensus.cc:504] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.385289 7725 leader_election.cc:288] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.385816 5839 raft_consensus.cc:348] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.386102 5839 raft_consensus.cc:374] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.386183 5839 raft_consensus.cc:729] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.386790 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.388185 5841 ts_tablet_manager.cc:1888] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.388327 5841 log.cc:1196] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/47a154769d0f4501837888feba98f33d
W20240507 08:12:20.388535 5378 leader_election.cc:341] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.388746 5841 ts_tablet_manager.cc:1909] T 47a154769d0f4501837888feba98f33d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.389043 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1247) took 3448 ms. Trace:
I20240507 08:12:20.389120 5841 rpcz_store.cc:276] 0507 08:12:16.940975 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.942088 (+ 1113us) service_pool.cc:225] Handling call
0507 08:12:20.359818 (+3417730us) ts_tablet_manager.cc:1120] Deleting tablet 47a154769d0f4501837888feba98f33d
0507 08:12:20.359827 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.365010 (+ 5183us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.387493 (+ 22483us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.389027 (+ 1534us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3406441}
I20240507 08:12:20.389480 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 47a154769d0f4501837888feba98f33d (table default.Table_3a7027841ed1443096999162f8cdd1ea [id=567b19917c9148d4ad6ff350294bf568]) successfully deleted
I20240507 08:12:20.390169 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1f3f2ed28af24116b1d81eecee74897c" } modify_external_catalogs: true
I20240507 08:12:20.390328 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_1f3f2ed28af24116b1d81eecee74897c" } modify_external_catalogs: true
I20240507 08:12:20.390544 5843 tablet_bootstrap.cc:492] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.390446 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_95e928fd53d04396b4e7a984096d5853"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.390652 5843 ts_tablet_manager.cc:1367] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.108s user 0.001s sys 0.001s
I20240507 08:12:20.391165 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c4a924e93759482bba1dfca1998c326c" } modify_external_catalogs: true
I20240507 08:12:20.391294 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_c4a924e93759482bba1dfca1998c326c" } modify_external_catalogs: true
I20240507 08:12:20.391292 5843 raft_consensus.cc:348] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.391469 5843 raft_consensus.cc:374] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.391532 5843 raft_consensus.cc:729] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
W20240507 08:12:20.391608 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95e928fd53d04396b4e7a984096d5853 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.391732 5843 consensus_queue.cc:260] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.394699 6513 ts_tablet_manager.cc:1875] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.397068 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1253) took 3335 ms. Trace:
I20240507 08:12:20.397205 5841 rpcz_store.cc:276] 0507 08:12:17.061703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.061760 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:20.396951 (+3335191us) ts_tablet_manager.cc:1120] Deleting tablet c1b0a2dddb15451a8e6230ffb52d5302
0507 08:12:20.396961 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.397050 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3334980}
W20240507 08:12:20.398054 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c1b0a2dddb15451a8e6230ffb52d5302 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b0a2dddb15451a8e6230ffb52d5302 already in progress: creating tablet
I20240507 08:12:20.398380 5841 tablet_replica.cc:330] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.398577 5841 raft_consensus.cc:2229] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.398770 5841 raft_consensus.cc:2258] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.399286 5841 ts_tablet_manager.cc:1875] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.399549 5263 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cd9885f1f58049c599b7710b4e1f8977" } modify_external_catalogs: true
I20240507 08:12:20.399778 5263 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_cd9885f1f58049c599b7710b4e1f8977" } modify_external_catalogs: true
I20240507 08:12:20.400374 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ee86ace127244a70915ba25f74576527"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.400966 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee86ace127244a70915ba25f74576527 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.406186 7725 raft_consensus.cc:482] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.406340 7725 raft_consensus.cc:504] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.407313 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b967f56b73aa4c4eb838c42c831b9712 consensus state has no leader
I20240507 08:12:20.407371 7725 leader_election.cc:288] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.409416 5843 ts_tablet_manager.cc:1398] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20240507 08:12:20.409785 5843 tablet_bootstrap.cc:492] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.410977 7720 raft_consensus.cc:482] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.411119 7720 raft_consensus.cc:504] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.411964 7720 leader_election.cc:288] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.412550 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ccdad494794192b3c37c8d5cbfcb17" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.412775 5520 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:20.412866 5841 ts_tablet_manager.cc:1888] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:20.413013 5841 log.cc:1196] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e17502bb4ac64f6ca9c77a3b00ab9f4a
I20240507 08:12:20.413391 5841 ts_tablet_manager.cc:1909] T e17502bb4ac64f6ca9c77a3b00ab9f4a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:20.413786 5378 leader_election.cc:341] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.414140 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1254) took 3333 ms. Trace:
I20240507 08:12:20.414220 5841 rpcz_store.cc:276] 0507 08:12:17.080304 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.080361 (+ 57us) service_pool.cc:225] Handling call
0507 08:12:20.398339 (+3317978us) ts_tablet_manager.cc:1120] Deleting tablet e17502bb4ac64f6ca9c77a3b00ab9f4a
0507 08:12:20.398348 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.402239 (+ 3891us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.404995 (+ 2756us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.414124 (+ 9129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3317788}
I20240507 08:12:20.414538 5841 tablet_replica.cc:330] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.414695 5841 raft_consensus.cc:2229] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.414917 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e17502bb4ac64f6ca9c77a3b00ab9f4a (table default.Table_d43fc2157a1546e39dc17b5f26047574 [id=746de1d0664945fd9a4a2cf4bed5405a]) successfully deleted
I20240507 08:12:20.414997 5841 raft_consensus.cc:2258] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.415493 5841 ts_tablet_manager.cc:1875] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.421550 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_92a943da9da941f9aa6f4dd810a2ec6d" } new_table_name: "default.Table_c005730e806249b7b5fb8a0dfab66f9c" modify_external_catalogs: true
I20240507 08:12:20.427184 7720 raft_consensus.cc:482] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.427338 7720 raft_consensus.cc:504] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.428071 7720 leader_election.cc:288] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.429144 5843 tablet_bootstrap.cc:654] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.432258 7579 raft_consensus.cc:3037] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.432143 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3b455b2145f64d548cfd91cd286cdbe6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.432930 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b455b2145f64d548cfd91cd286cdbe6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:20.433658 7741 log.cc:924] Time spent T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
W20240507 08:12:20.433789 5520 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: cd4379b6fded4a52843f55f27ae67121
W20240507 08:12:20.434154 7742 log.cc:924] Time spent T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20240507 08:12:20.434372 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.434983 5377 leader_election.cc:302] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.435403 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3ccc3e67eeb34c42b36edde7bacd23a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.435497 7725 raft_consensus.cc:2781] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.435564 7725 raft_consensus.cc:482] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.435618 7725 raft_consensus.cc:3037] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.435657 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.435958 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f0b399a2d074f4ca3626ff17bd5518e" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.436434 5718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ccdad494794192b3c37c8d5cbfcb17" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.436640 5718 raft_consensus.cc:2454] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.436924 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.437103 5720 raft_consensus.cc:2454] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.438006 7725 raft_consensus.cc:504] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.438050 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 4e5439844f384513b4719cfe4f4e5d9b (table default.Table_9342125a817c439eb044fbe82437c21a [id=64f54c3a3fba416f8810335c6f651ced]) successfully deleted
I20240507 08:12:20.441905 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eec3704b135740f1a906d60f3ea87ffe" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.443068 7745 log.cc:924] Time spent T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20240507 08:12:20.443143 7079 ts_tablet_manager.cc:1888] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.443137 7725 leader_election.cc:288] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.443238 7079 log.cc:1196] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/a56cb45381604ac9a69e6f3edaa4eefc
I20240507 08:12:20.443359 5583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.443574 7079 ts_tablet_manager.cc:1909] T a56cb45381604ac9a69e6f3edaa4eefc P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:20.448702 6513 ts_tablet_manager.cc:1888] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.448796 6513 log.cc:1196] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e14487e625374060b118c9cce4500b3f
W20240507 08:12:20.448828 5378 leader_election.cc:341] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.449107 5517 leader_election.cc:341] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 85dc58133d2b4f5496b4e570ae6494c0
I20240507 08:12:20.449111 6513 ts_tablet_manager.cc:1909] T e14487e625374060b118c9cce4500b3f P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
W20240507 08:12:20.449156 5654 leader_election.cc:341] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.449205 5517 leader_election.cc:302] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
W20240507 08:12:20.449425 5654 leader_election.cc:341] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.449559 7720 raft_consensus.cc:2726] T 85dc58133d2b4f5496b4e570ae6494c0 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:20.450085 5377 leader_election.cc:341] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.450177 5377 leader_election.cc:302] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.450485 5377 leader_election.cc:302] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.450752 5377 leader_election.cc:302] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.451056 7499 raft_consensus.cc:2726] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.451227 7499 raft_consensus.cc:2781] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.451283 7499 raft_consensus.cc:482] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.451339 7499 raft_consensus.cc:3037] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.449426 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1205) took 4683 ms. Trace:
I20240507 08:12:20.452003 5840 tablet_bootstrap.cc:492] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.452144 5840 ts_tablet_manager.cc:1367] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.002s
I20240507 08:12:20.452065 6513 rpcz_store.cc:276] 0507 08:12:15.766096 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.766940 (+ 844us) service_pool.cc:225] Handling call
0507 08:12:20.288677 (+4521737us) ts_tablet_manager.cc:1120] Deleting tablet e14487e625374060b118c9cce4500b3f
0507 08:12:20.288686 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.445986 (+157300us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.448623 (+ 2637us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.449411 (+ 788us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4507038}
I20240507 08:12:20.452417 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1203) took 4686 ms. Trace:
I20240507 08:12:20.452482 6513 rpcz_store.cc:276] 0507 08:12:15.765837 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.799267 (+ 33430us) service_pool.cc:225] Handling call
0507 08:12:20.452338 (+4653071us) ts_tablet_manager.cc:1120] Deleting tablet 28e5a380f2784e3a806afcbb5cd2519a
0507 08:12:20.452347 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.452406 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4652901}
I20240507 08:12:20.452765 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1232) took 4567 ms. Trace:
I20240507 08:12:20.452829 5840 raft_consensus.cc:348] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.453033 5840 raft_consensus.cc:374] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.453075 5840 raft_consensus.cc:729] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.453248 5840 consensus_queue.cc:260] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.386618 5839 consensus_queue.cc:260] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.453677 7378 raft_consensus.cc:2781] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.453768 7378 raft_consensus.cc:482] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.453835 7378 raft_consensus.cc:3037] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.455829 7750 tablet.cc:1722] T 7e3e8b4d5d594f378451009c313f6bfa P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:20.456890 5839 ts_tablet_manager.cc:1398] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.072s user 0.003s sys 0.000s
I20240507 08:12:20.457824 5840 ts_tablet_manager.cc:1398] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:20.458087 5840 tablet_bootstrap.cc:492] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.459338 5840 tablet_bootstrap.cc:654] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.461515 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
W20240507 08:12:20.462438 5517 leader_election.cc:341] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: cd4379b6fded4a52843f55f27ae67121
I20240507 08:12:20.462570 5517 leader_election.cc:302] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.462913 7720 raft_consensus.cc:2726] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.464306 7499 raft_consensus.cc:504] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.464829 7499 leader_election.cc:288] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.465858 7748 raft_consensus.cc:482] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.466010 7748 raft_consensus.cc:504] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.466801 7748 leader_election.cc:288] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:20.467713 5654 leader_election.cc:341] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.467962 7378 raft_consensus.cc:504] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.468080 5654 leader_election.cc:341] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.468310 5654 consensus_peers.cc:485] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:20.468425 5654 consensus_peers.cc:485] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.468459 7378 leader_election.cc:288] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.468737 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet e14487e625374060b118c9cce4500b3f (table default.Table_5efeda526693400790a6dfb0d118fac0 [id=8adafd6d08374df7a4f0fa1fd90991fe]) successfully deleted
W20240507 08:12:20.469208 5248 catalog_manager.cc:4797] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 28e5a380f2784e3a806afcbb5cd2519a because the tablet was not found. No further retry: Not found: Tablet not found: 28e5a380f2784e3a806afcbb5cd2519a
I20240507 08:12:20.469506 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.469704 5445 raft_consensus.cc:2454] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
W20240507 08:12:20.469857 5495 tablet.cc:2369] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240507 08:12:20.470054 5656 leader_election.cc:302] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.471421 7740 raft_consensus.cc:2781] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.471539 7740 raft_consensus.cc:482] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.472241 7740 raft_consensus.cc:3037] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.474846 7579 raft_consensus.cc:504] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.475380 7740 raft_consensus.cc:504] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.475804 7579 leader_election.cc:288] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.475849 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b6a2f4aa14f2a927fa9ad56a62a5a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.476033 5442 raft_consensus.cc:3037] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.476481 7740 leader_election.cc:288] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.476817 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.476997 5445 raft_consensus.cc:3037] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.479044 5840 tablet_bootstrap.cc:492] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.479159 5840 ts_tablet_manager.cc:1367] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.004s
I20240507 08:12:20.479487 5445 raft_consensus.cc:2454] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.479774 7358 raft_consensus.cc:482] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.479835 5656 leader_election.cc:302] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.479885 7358 raft_consensus.cc:504] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.480214 5840 raft_consensus.cc:348] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.480370 5840 raft_consensus.cc:374] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.480427 5840 raft_consensus.cc:729] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.480835 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.481046 5445 raft_consensus.cc:2454] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.481882 5261 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:20.482393 5442 raft_consensus.cc:2454] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.483325 7649 raft_consensus.cc:2781] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.480504 7358 leader_election.cc:288] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.481379 5656 leader_election.cc:302] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.483420 5840 consensus_queue.cc:260] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.483647 7649 raft_consensus.cc:686] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.483781 5656 leader_election.cc:302] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.483995 7656 raft_consensus.cc:2781] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.483909 7649 consensus_queue.cc:237] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.484067 7743 raft_consensus.cc:2781] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.484184 7743 raft_consensus.cc:686] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.484381 5840 ts_tablet_manager.cc:1398] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240507 08:12:20.452824 6513 rpcz_store.cc:276] 0507 08:12:15.885216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.885396 (+ 180us) service_pool.cc:225] Handling call
0507 08:12:20.452680 (+4567284us) ts_tablet_manager.cc:1120] Deleting tablet 0ddb7dd28f43488581d3a75bf1d68a8a
0507 08:12:20.452688 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.452754 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4567104}
I20240507 08:12:20.484407 7743 consensus_queue.cc:237] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.484676 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1233) took 4599 ms. Trace:
I20240507 08:12:20.484782 6513 rpcz_store.cc:276] 0507 08:12:15.885355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.885647 (+ 292us) service_pool.cc:225] Handling call
0507 08:12:20.484566 (+4598919us) ts_tablet_manager.cc:1120] Deleting tablet b15c8ad2a8324c75a1ef49e219b962cf
0507 08:12:20.484575 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.484660 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4598606}
I20240507 08:12:20.485129 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1234) took 4551 ms. Trace:
I20240507 08:12:20.485189 6513 rpcz_store.cc:276] 0507 08:12:15.934036 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.934103 (+ 67us) service_pool.cc:225] Handling call
0507 08:12:20.485039 (+4550936us) ts_tablet_manager.cc:1120] Deleting tablet c20c60fe0aad4cb3b12d04ad1a87d9f4
0507 08:12:20.485047 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.485117 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550748}
I20240507 08:12:20.485374 6513 tablet_replica.cc:330] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.485543 6513 raft_consensus.cc:2229] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.484056 7656 raft_consensus.cc:482] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.485679 7656 raft_consensus.cc:3037] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.485824 6513 raft_consensus.cc:2258] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.486289 6513 ts_tablet_manager.cc:1875] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.488399 7656 raft_consensus.cc:504] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.489470 7656 leader_election.cc:288] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.490061 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.490582 5442 raft_consensus.cc:3037] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.496160 7579 raft_consensus.cc:482] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.496316 7579 raft_consensus.cc:504] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.497066 7579 leader_election.cc:288] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.497224 5442 raft_consensus.cc:2454] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.499228 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.499393 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ccdad494794192b3c37c8d5cbfcb17" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.499788 5840 tablet_bootstrap.cc:492] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.499924 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.500401 5839 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.501626 5840 tablet_bootstrap.cc:654] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.501345 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1245) took 2666 ms. Trace:
I20240507 08:12:20.501778 7079 rpcz_store.cc:276] 0507 08:12:17.834810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.854234 (+ 19424us) service_pool.cc:225] Handling call
0507 08:12:20.361761 (+2507527us) ts_tablet_manager.cc:1120] Deleting tablet a56cb45381604ac9a69e6f3edaa4eefc
0507 08:12:20.361769 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.425177 (+ 63408us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.443050 (+ 17873us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.501328 (+ 58278us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507304}
I20240507 08:12:20.502271 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1250) took 2541 ms. Trace:
I20240507 08:12:20.502404 7079 rpcz_store.cc:276] 0507 08:12:17.960734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.960785 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:20.502164 (+2541379us) ts_tablet_manager.cc:1120] Deleting tablet d718aa8fe99041688964d1a05cef49a0
0507 08:12:20.502173 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.502257 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2541153}
I20240507 08:12:20.503988 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1255) took 2469 ms. Trace:
I20240507 08:12:20.504122 7079 rpcz_store.cc:276] 0507 08:12:18.034707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.034755 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:20.502685 (+2467930us) ts_tablet_manager.cc:1120] Deleting tablet cd4d1fa6a8d94af982e1d23c08cbdeac
0507 08:12:20.503864 (+ 1179us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.503971 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467739}
I20240507 08:12:20.504499 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1256) took 2350 ms. Trace:
I20240507 08:12:20.504619 7079 rpcz_store.cc:276] 0507 08:12:18.153916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.154686 (+ 770us) service_pool.cc:225] Handling call
0507 08:12:20.504404 (+2349718us) ts_tablet_manager.cc:1120] Deleting tablet 8d1a4e05273b4f8c83dc8c7825a39310
0507 08:12:20.504416 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.504486 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349520}
I20240507 08:12:20.505026 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1265) took 2329 ms. Trace:
I20240507 08:12:20.505153 7079 rpcz_store.cc:276] 0507 08:12:18.175768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.176300 (+ 532us) service_pool.cc:225] Handling call
0507 08:12:20.504927 (+2328627us) ts_tablet_manager.cc:1120] Deleting tablet f859146c376c4b1eb52bff953f05837b
0507 08:12:20.504935 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.505012 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2328468}
I20240507 08:12:20.506436 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet a56cb45381604ac9a69e6f3edaa4eefc (table default.Table_43a710eac1c44c56a13a40993d865715 [id=4ed1bd67ea9241459a269a6d34b93430]) successfully deleted
W20240507 08:12:20.506750 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d718aa8fe99041688964d1a05cef49a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d718aa8fe99041688964d1a05cef49a0 already in progress: creating tablet
W20240507 08:12:20.507107 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet cd4d1fa6a8d94af982e1d23c08cbdeac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4d1fa6a8d94af982e1d23c08cbdeac already in progress: creating tablet
W20240507 08:12:20.507447 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 8d1a4e05273b4f8c83dc8c7825a39310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1a4e05273b4f8c83dc8c7825a39310 already in progress: creating tablet
W20240507 08:12:20.507715 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet f859146c376c4b1eb52bff953f05837b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f859146c376c4b1eb52bff953f05837b already in progress: creating tablet
I20240507 08:12:20.497300 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.509004 5445 raft_consensus.cc:2454] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.497695 5843 tablet_bootstrap.cc:492] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:20.499751 5378 leader_election.cc:341] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.509315 5656 leader_election.cc:302] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
W20240507 08:12:20.509449 5378 leader_election.cc:341] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.509562 7579 raft_consensus.cc:2781] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.509557 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1266) took 2333 ms. Trace:
I20240507 08:12:20.509692 7579 raft_consensus.cc:482] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.509660 7079 rpcz_store.cc:276] 0507 08:12:18.175885 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.177159 (+ 1274us) service_pool.cc:225] Handling call
0507 08:12:20.505401 (+2328242us) ts_tablet_manager.cc:1120] Deleting tablet 636e8f123b9540f3afedf02e4df5dfb5
0507 08:12:20.505615 (+ 214us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.509542 (+ 3927us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2310720}
I20240507 08:12:20.509802 5843 ts_tablet_manager.cc:1367] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.100s user 0.004s sys 0.000s
I20240507 08:12:20.509900 7579 raft_consensus.cc:3037] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.510000 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1299) took 1915 ms. Trace:
W20240507 08:12:20.510079 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 636e8f123b9540f3afedf02e4df5dfb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 636e8f123b9540f3afedf02e4df5dfb5 already in progress: creating tablet
I20240507 08:12:20.510076 7079 rpcz_store.cc:276] 0507 08:12:18.594959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.652710 (+ 57751us) service_pool.cc:225] Handling call
0507 08:12:20.509882 (+1857172us) ts_tablet_manager.cc:1120] Deleting tablet 04a14cf04d7a4ce3a2df939d0db515b4
0507 08:12:20.509891 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.509984 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1854320}
I20240507 08:12:20.510361 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1301) took 1785 ms. Trace:
W20240507 08:12:20.510471 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 04a14cf04d7a4ce3a2df939d0db515b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04a14cf04d7a4ce3a2df939d0db515b4 already in progress: creating tablet
I20240507 08:12:20.510439 7079 rpcz_store.cc:276] 0507 08:12:18.725247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.729771 (+ 4524us) service_pool.cc:225] Handling call
0507 08:12:20.510276 (+1780505us) ts_tablet_manager.cc:1120] Deleting tablet ffc89e00efaa4bde9242be40cf409c59
0507 08:12:20.510283 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.510348 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1780027}
I20240507 08:12:20.510565 5843 raft_consensus.cc:348] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.510759 5843 raft_consensus.cc:374] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.510763 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1302) took 1780 ms. Trace:
W20240507 08:12:20.510829 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ffc89e00efaa4bde9242be40cf409c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffc89e00efaa4bde9242be40cf409c59 already in progress: creating tablet
I20240507 08:12:20.510859 5843 raft_consensus.cc:729] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.510840 7079 rpcz_store.cc:276] 0507 08:12:18.730229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.736392 (+ 6163us) service_pool.cc:225] Handling call
0507 08:12:20.510672 (+1774280us) ts_tablet_manager.cc:1120] Deleting tablet 120bce4b20714fc1a0e1237abc00d5bb
0507 08:12:20.510680 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.510749 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1774116}
I20240507 08:12:20.511135 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1300) took 1786 ms. Trace:
W20240507 08:12:20.511183 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 120bce4b20714fc1a0e1237abc00d5bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120bce4b20714fc1a0e1237abc00d5bb already in progress: creating tablet
I20240507 08:12:20.511099 5843 consensus_queue.cc:260] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.511202 7079 rpcz_store.cc:276] 0507 08:12:18.724779 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.782597 (+ 57818us) service_pool.cc:225] Handling call
0507 08:12:20.511049 (+1728452us) ts_tablet_manager.cc:1120] Deleting tablet 5aca2077b76b49e3bf9daae57dc09b8f
0507 08:12:20.511056 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.511122 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1726327}
I20240507 08:12:20.511505 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1304) took 1651 ms. Trace:
I20240507 08:12:20.511572 7079 rpcz_store.cc:276] 0507 08:12:18.859905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.860086 (+ 181us) service_pool.cc:225] Handling call
0507 08:12:20.511400 (+1651314us) ts_tablet_manager.cc:1120] Deleting tablet fdffec6ce0da4c409466783bd76a6b4b
0507 08:12:20.511408 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.511491 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651154}
W20240507 08:12:20.511651 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 5aca2077b76b49e3bf9daae57dc09b8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5aca2077b76b49e3bf9daae57dc09b8f already in progress: creating tablet
I20240507 08:12:20.511734 5843 ts_tablet_manager.cc:1398] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:20.511799 7079 tablet_replica.cc:330] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:20.511968 7079 raft_consensus.cc:2229] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:20.511976 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet fdffec6ce0da4c409466783bd76a6b4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdffec6ce0da4c409466783bd76a6b4b already in progress: creating tablet
I20240507 08:12:20.512109 7079 raft_consensus.cc:2258] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.512657 7079 ts_tablet_manager.cc:1875] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.513244 5839 tablet_bootstrap.cc:654] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.513670 5656 leader_election.cc:302] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.513971 7656 raft_consensus.cc:2781] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.514010 7579 raft_consensus.cc:504] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.516422 5843 tablet_bootstrap.cc:492] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.501454 5841 ts_tablet_manager.cc:1888] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.517748 5841 log.cc:1196] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/2bb648fe98e74248b5cd65603de58a02
I20240507 08:12:20.518193 5841 ts_tablet_manager.cc:1909] T 2bb648fe98e74248b5cd65603de58a02 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.518821 5843 tablet_bootstrap.cc:654] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.514120 7656 raft_consensus.cc:686] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.523870 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b13fbddcab5460db69f43e63f490fba" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.525782 5720 raft_consensus.cc:3037] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.525874 7656 consensus_queue.cc:237] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.526516 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8b47c57b0f964c7894461a594bb22539
I20240507 08:12:20.527611 5259 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_97792a6d740f4599b202087719f29978" } modify_external_catalogs: true
I20240507 08:12:20.527753 5259 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_97792a6d740f4599b202087719f29978" } modify_external_catalogs: true
I20240507 08:12:20.528285 5720 raft_consensus.cc:2454] T 2b13fbddcab5460db69f43e63f490fba P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.524062 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63ccdad494794192b3c37c8d5cbfcb17" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.528518 5717 raft_consensus.cc:3037] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.529593 7579 leader_election.cc:288] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.529994 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b47c57b0f964c7894461a594bb22539 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.531462 5717 raft_consensus.cc:2454] T 63ccdad494794192b3c37c8d5cbfcb17 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.520190 7720 raft_consensus.cc:482] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.531699 7720 raft_consensus.cc:504] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.532388 7720 leader_election.cc:288] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.532389 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.524230 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5976eca006448e0b9f2573acd738bb1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.532562 5713 raft_consensus.cc:3037] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.533031 5442 raft_consensus.cc:3037] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
W20240507 08:12:20.533032 5520 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:12:20.537612 5713 raft_consensus.cc:2454] T a5976eca006448e0b9f2573acd738bb1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
W20240507 08:12:20.538514 5517 leader_election.cc:341] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 831153d2b89e43f8b08b086f7a8f3a54
I20240507 08:12:20.538615 5517 leader_election.cc:302] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.538980 5843 tablet_bootstrap.cc:492] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.538987 7720 raft_consensus.cc:2726] T 831153d2b89e43f8b08b086f7a8f3a54 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.539796 5840 tablet_bootstrap.cc:492] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.540009 5843 ts_tablet_manager.cc:1367] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20240507 08:12:20.540212 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b47c57b0f964c7894461a594bb22539 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.540735 5257 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 645ae26bb26f43d298afe98817d88fa4 (table default.Table_1cb491139e6446b19eef66a5838468a7 [id=482211c676ea45a681f5a8bcd7cdb8a3]). Expected version 3 got 0
I20240507 08:12:20.541463 5840 ts_tablet_manager.cc:1367] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.001s
I20240507 08:12:20.542207 5840 raft_consensus.cc:348] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.542371 5840 raft_consensus.cc:374] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.542089 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_04f2b747a43744628a7af8d9f4ea5258"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.543983 5843 raft_consensus.cc:348] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.523885 6513 ts_tablet_manager.cc:1888] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20240507 08:12:20.544232 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04f2b747a43744628a7af8d9f4ea5258 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.544459 5843 raft_consensus.cc:374] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.544636 6513 log.cc:1196] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c839c8a5041f4c3da9bc0749afee1413
I20240507 08:12:20.544721 5840 raft_consensus.cc:729] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.545521 5840 consensus_queue.cc:260] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.548256 5580 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.548403 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "156bf04b57724134a45fa9fc477596c5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.545217 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1255) took 3451 ms. Trace:
I20240507 08:12:20.548720 5582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.548858 5841 rpcz_store.cc:276] 0507 08:12:17.093906 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.093962 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:20.414483 (+3320521us) ts_tablet_manager.cc:1120] Deleting tablet 2bb648fe98e74248b5cd65603de58a02
0507 08:12:20.414491 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.467047 (+ 52556us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.501327 (+ 34280us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.545173 (+ 43846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3320323}
I20240507 08:12:20.549244 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6def4bf6a7554a4090a39b1823c4097c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.549471 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1256) took 3300 ms. Trace:
I20240507 08:12:20.549525 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 2bb648fe98e74248b5cd65603de58a02 (table default.Table_b19f64f84d434d8684f3b784794f2f52 [id=58501826cb5744918aba9bf338a0eccf]) successfully deleted
I20240507 08:12:20.549559 5841 rpcz_store.cc:276] 0507 08:12:17.248566 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.248835 (+ 269us) service_pool.cc:225] Handling call
0507 08:12:20.549336 (+3300501us) ts_tablet_manager.cc:1120] Deleting tablet fda4b1ceaf0b4f3385258f3703e4461b
0507 08:12:20.549345 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.549428 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3300323}
W20240507 08:12:20.549948 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 0ddb7dd28f43488581d3a75bf1d68a8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ddb7dd28f43488581d3a75bf1d68a8a already in progress: creating tablet
W20240507 08:12:20.550148 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet b15c8ad2a8324c75a1ef49e219b962cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b15c8ad2a8324c75a1ef49e219b962cf already in progress: creating tablet
I20240507 08:12:20.550259 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1257) took 3158 ms. Trace:
W20240507 08:12:20.550349 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c20c60fe0aad4cb3b12d04ad1a87d9f4 already in progress: creating tablet
I20240507 08:12:20.550354 5841 rpcz_store.cc:276] 0507 08:12:17.391879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.392072 (+ 193us) service_pool.cc:225] Handling call
0507 08:12:20.550079 (+3158007us) ts_tablet_manager.cc:1120] Deleting tablet 3eb49743134b4f84aa6fdd9760c13504
0507 08:12:20.550087 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.550159 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3157819}
W20240507 08:12:20.550700 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet fda4b1ceaf0b4f3385258f3703e4461b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fda4b1ceaf0b4f3385258f3703e4461b already in progress: creating tablet
I20240507 08:12:20.550767 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1258) took 3158 ms. Trace:
I20240507 08:12:20.550837 5841 rpcz_store.cc:276] 0507 08:12:17.392020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.392293 (+ 273us) service_pool.cc:225] Handling call
0507 08:12:20.550659 (+3158366us) ts_tablet_manager.cc:1120] Deleting tablet 5481ddbcd31449c4b44afb87a97138bd
0507 08:12:20.550667 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.550735 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3158245}
W20240507 08:12:20.550895 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3eb49743134b4f84aa6fdd9760c13504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3eb49743134b4f84aa6fdd9760c13504 already in progress: creating tablet
W20240507 08:12:20.551079 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5481ddbcd31449c4b44afb87a97138bd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5481ddbcd31449c4b44afb87a97138bd already in progress: creating tablet
I20240507 08:12:20.551329 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1259) took 3149 ms. Trace:
I20240507 08:12:20.551380 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d17b6a2f4aa14f2a927fa9ad56a62a5a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.551538 5841 rpcz_store.cc:276] 0507 08:12:17.402227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.402275 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:20.551118 (+3148843us) ts_tablet_manager.cc:1120] Deleting tablet 71fcb6a8839840838bedb777ff1cbbcc
0507 08:12:20.551125 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.551190 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148686}
W20240507 08:12:20.551986 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 71fcb6a8839840838bedb777ff1cbbcc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71fcb6a8839840838bedb777ff1cbbcc already in progress: creating tablet
I20240507 08:12:20.552021 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1260) took 3090 ms. Trace:
I20240507 08:12:20.552103 5841 rpcz_store.cc:276] 0507 08:12:17.461478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.461537 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:20.551914 (+3090377us) ts_tablet_manager.cc:1120] Deleting tablet 3ef4375925474e789a39a2c28fa6adfe
0507 08:12:20.551923 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.551988 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3090178}
W20240507 08:12:20.552419 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3ef4375925474e789a39a2c28fa6adfe because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3ef4375925474e789a39a2c28fa6adfe already in progress: creating tablet
I20240507 08:12:20.552526 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1261) took 2948 ms. Trace:
I20240507 08:12:20.552652 5841 rpcz_store.cc:276] 0507 08:12:17.604540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.604587 (+ 47us) service_pool.cc:225] Handling call
0507 08:12:20.552405 (+2947818us) ts_tablet_manager.cc:1120] Deleting tablet c9a65a8e939f4b7d89d00720845ea1fa
0507 08:12:20.552412 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.552494 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2947629}
W20240507 08:12:20.553036 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c9a65a8e939f4b7d89d00720845ea1fa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9a65a8e939f4b7d89d00720845ea1fa already in progress: creating tablet
I20240507 08:12:20.553143 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1270) took 2896 ms. Trace:
I20240507 08:12:20.553223 5841 rpcz_store.cc:276] 0507 08:12:17.656672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.656732 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:20.553031 (+2896299us) ts_tablet_manager.cc:1120] Deleting tablet d8e03f8655714d309d5c8ae49c4b314b
0507 08:12:20.553040 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.553110 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896114}
I20240507 08:12:20.553711 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1269) took 2897 ms. Trace:
W20240507 08:12:20.553884 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d8e03f8655714d309d5c8ae49c4b314b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8e03f8655714d309d5c8ae49c4b314b already in progress: creating tablet
I20240507 08:12:20.553857 5841 rpcz_store.cc:276] 0507 08:12:17.656577 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.656848 (+ 271us) service_pool.cc:225] Handling call
0507 08:12:20.553600 (+2896752us) ts_tablet_manager.cc:1120] Deleting tablet 4b78c08d758043c78d5bb8281b7f2d62
0507 08:12:20.553608 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.553672 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2896582}
W20240507 08:12:20.554212 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4b78c08d758043c78d5bb8281b7f2d62 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b78c08d758043c78d5bb8281b7f2d62 already in progress: creating tablet
I20240507 08:12:20.554277 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1271) took 2897 ms. Trace:
I20240507 08:12:20.554351 5841 rpcz_store.cc:276] 0507 08:12:17.656791 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.656954 (+ 163us) service_pool.cc:225] Handling call
0507 08:12:20.554166 (+2897212us) ts_tablet_manager.cc:1120] Deleting tablet fe721ba2f0ed4a3a887e7496e85d163c
0507 08:12:20.554174 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.554242 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897013}
I20240507 08:12:20.554756 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1278) took 2861 ms. Trace:
I20240507 08:12:20.554839 5841 rpcz_store.cc:276] 0507 08:12:17.692953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.693012 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:20.554645 (+2861633us) ts_tablet_manager.cc:1120] Deleting tablet cc3d1e9e7e53483fa75f7cd918295e50
0507 08:12:20.554653 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.554713 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2861484}
W20240507 08:12:20.555127 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet fe721ba2f0ed4a3a887e7496e85d163c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe721ba2f0ed4a3a887e7496e85d163c already in progress: creating tablet
I20240507 08:12:20.555248 5841 tablet_replica.cc:330] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:20.555348 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cc3d1e9e7e53483fa75f7cd918295e50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc3d1e9e7e53483fa75f7cd918295e50 already in progress: creating tablet
I20240507 08:12:20.555596 5843 raft_consensus.cc:729] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.555881 5843 consensus_queue.cc:260] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.557104 5840 ts_tablet_manager.cc:1398] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.016s user 0.001s sys 0.000s
I20240507 08:12:20.557608 5843 ts_tablet_manager.cc:1398] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240507 08:12:20.557896 5840 tablet_bootstrap.cc:492] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.556535 6513 ts_tablet_manager.cc:1909] T c839c8a5041f4c3da9bc0749afee1413 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.558907 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1235) took 4624 ms. Trace:
I20240507 08:12:20.559057 6513 rpcz_store.cc:276] 0507 08:12:15.934190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.934328 (+ 138us) service_pool.cc:225] Handling call
0507 08:12:20.485342 (+4551014us) ts_tablet_manager.cc:1120] Deleting tablet c839c8a5041f4c3da9bc0749afee1413
0507 08:12:20.485349 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.520919 (+ 35570us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.523802 (+ 2883us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.558883 (+ 35081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4550889}
I20240507 08:12:20.559512 6513 tablet_replica.cc:330] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.559868 6513 raft_consensus.cc:2229] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.561022 6513 raft_consensus.cc:2258] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.561684 6513 ts_tablet_manager.cc:1875] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:20.563508 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b95711fb20684223a02b09114419ce9a consensus state has no leader
I20240507 08:12:20.563877 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_17597b6b0b0f401b920b72234906890c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.564368 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17597b6b0b0f401b920b72234906890c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.564532 5840 tablet_bootstrap.cc:654] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.565801 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee96356f684941cc995051508593ecd8" } modify_external_catalogs: true
I20240507 08:12:20.565951 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_ee96356f684941cc995051508593ecd8" } modify_external_catalogs: true
I20240507 08:12:20.566934 5264 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3dc2eefb649e40c5bc0e3d3c61eb2f46" } new_table_name: "default.Table_da2e23a62bda4d63a6db298e0ccbf886" modify_external_catalogs: true
I20240507 08:12:20.568145 5442 raft_consensus.cc:2454] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.568377 5843 tablet_bootstrap.cc:492] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.568341 5841 raft_consensus.cc:2229] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.568516 5656 leader_election.cc:302] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.568876 5841 raft_consensus.cc:2258] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.568888 7743 raft_consensus.cc:2781] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.569020 7743 raft_consensus.cc:686] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.569532 5841 ts_tablet_manager.cc:1875] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.569900 5843 tablet_bootstrap.cc:654] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.569242 7743 consensus_queue.cc:237] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.565327 5654 leader_election.cc:341] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.574218 5654 leader_election.cc:341] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.574554 5654 leader_election.cc:341] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.574856 5654 leader_election.cc:341] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.575145 5654 leader_election.cc:341] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:20.575511 5654 leader_election.cc:341] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.576236 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c839c8a5041f4c3da9bc0749afee1413 (table default.Table_e19ac67deb044b61b0e6194e8b54aae0 [id=9c2cb8887f5344ff8a6d58b3f7a72459]) successfully deleted
I20240507 08:12:20.576900 5839 tablet_bootstrap.cc:492] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.577070 5839 ts_tablet_manager.cc:1367] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.001s
I20240507 08:12:20.577769 5839 raft_consensus.cc:348] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.577992 5839 raft_consensus.cc:374] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.578091 5839 raft_consensus.cc:729] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.578332 5839 consensus_queue.cc:260] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.578655 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c7f072f55a94cca938aa063637af9a5" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:20.578977 5839 ts_tablet_manager.cc:1398] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:20.576007 5377 leader_election.cc:302] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.580758 5377 leader_election.cc:302] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.581125 5377 leader_election.cc:302] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.581460 7499 raft_consensus.cc:2781] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.581681 7722 raft_consensus.cc:2781] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.581894 7499 raft_consensus.cc:686] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.582641 7499 consensus_queue.cc:237] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.583554 7725 raft_consensus.cc:2781] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.593492 5839 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.594161 5255 master_service.cc:946] ReplaceTablet: received request to replace tablet 310c601abead42e0bd2a2df1a0e16ec2 from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:20.595436 5839 tablet_bootstrap.cc:654] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.595435 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c3aa5dd8c7d44bb4944fcfeae3a249a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.596598 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c3aa5dd8c7d44bb4944fcfeae3a249a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.596925 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b47c57b0f964c7894461a594bb22539 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:20.597748 5654 leader_election.cc:341] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.601272 5840 tablet_bootstrap.cc:492] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.601418 5840 ts_tablet_manager.cc:1367] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
W20240507 08:12:20.601446 5264 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adfa454f8e2b4c2386c63c3cf765b5b1 consensus state has no leader
I20240507 08:12:20.601922 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2090ddab11e943099e287f1b2e71b6d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.602255 5840 raft_consensus.cc:348] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.602501 5840 raft_consensus.cc:374] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.602571 5840 raft_consensus.cc:729] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:20.602617 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2090ddab11e943099e287f1b2e71b6d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.602834 5840 consensus_queue.cc:260] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.603473 5840 ts_tablet_manager.cc:1398] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:20.605094 5840 tablet_bootstrap.cc:492] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.607450 5840 tablet_bootstrap.cc:654] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.611023 5841 ts_tablet_manager.cc:1888] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.611188 5841 log.cc:1196] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3c7f072f55a94cca938aa063637af9a5
I20240507 08:12:20.611317 5579 raft_consensus.cc:3037] T b9e715a29b0643b6ac61f9e73874c25c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.613431 5841 ts_tablet_manager.cc:1909] T 3c7f072f55a94cca938aa063637af9a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.613824 7378 raft_consensus.cc:482] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.613996 7378 raft_consensus.cc:504] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.614777 7378 leader_election.cc:288] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.615535 5585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605be6967a041388dfe81053fbb602e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.616117 5378 leader_election.cc:341] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.616783 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605be6967a041388dfe81053fbb602e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.616977 5717 raft_consensus.cc:2454] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.618852 5377 leader_election.cc:302] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.619438 7762 raft_consensus.cc:2781] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.619588 7762 raft_consensus.cc:482] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.619724 7762 raft_consensus.cc:3037] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.627730 6513 ts_tablet_manager.cc:1888] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.627849 6513 log.cc:1196] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/385f4c1eb7f4493a924b813734ddcfdb
W20240507 08:12:20.628067 5654 consensus_peers.cc:485] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.628196 6513 ts_tablet_manager.cc:1909] T 385f4c1eb7f4493a924b813734ddcfdb P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.628600 7722 raft_consensus.cc:686] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.628923 7722 consensus_queue.cc:237] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.633090 7725 raft_consensus.cc:686] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.636674 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1236) took 4702 ms. Trace:
I20240507 08:12:20.636729 5839 tablet_bootstrap.cc:492] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.636770 6513 rpcz_store.cc:276] 0507 08:12:15.934298 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:15.934483 (+ 185us) service_pool.cc:225] Handling call
0507 08:12:20.559462 (+4624979us) ts_tablet_manager.cc:1120] Deleting tablet 385f4c1eb7f4493a924b813734ddcfdb
0507 08:12:20.559474 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.586918 (+ 27444us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.627632 (+ 40714us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.636644 (+ 9012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4624856}
I20240507 08:12:20.636931 5839 ts_tablet_manager.cc:1367] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.001s
I20240507 08:12:20.637282 6513 tablet_replica.cc:330] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.637491 6513 raft_consensus.cc:2229] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.637504 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 385f4c1eb7f4493a924b813734ddcfdb (table default.Table_f46d6abd49364d9eb6fe1a78e3dc69a7 [id=a286976f28f7468784296bf1ada7bd29]) successfully deleted
I20240507 08:12:20.637817 6513 raft_consensus.cc:2258] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.638103 5839 raft_consensus.cc:348] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.638334 5839 raft_consensus.cc:374] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.638342 6513 ts_tablet_manager.cc:1875] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.638439 5839 raft_consensus.cc:729] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.638741 5839 consensus_queue.cc:260] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.640239 5839 ts_tablet_manager.cc:1398] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:20.640625 5839 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.613824 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1280) took 2920 ms. Trace:
I20240507 08:12:20.641719 7725 consensus_queue.cc:237] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.643309 5840 tablet_bootstrap.cc:492] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.643431 5840 ts_tablet_manager.cc:1367] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.002s
I20240507 08:12:20.643990 5840 raft_consensus.cc:348] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.644162 5840 raft_consensus.cc:374] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.644222 5840 raft_consensus.cc:729] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.644452 5840 consensus_queue.cc:260] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.644986 5840 ts_tablet_manager.cc:1398] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240507 08:12:20.645411 5839 tablet_bootstrap.cc:654] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.646543 5843 tablet_bootstrap.cc:492] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.646770 5843 ts_tablet_manager.cc:1367] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.003s
I20240507 08:12:20.647035 5840 tablet_bootstrap.cc:492] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.649930 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_81ffbde1da444f1fbd8963a91a87c8f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.650406 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81ffbde1da444f1fbd8963a91a87c8f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.653435 5841 rpcz_store.cc:276] 0507 08:12:17.693165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.693206 (+ 41us) service_pool.cc:225] Handling call
0507 08:12:20.555189 (+2861983us) ts_tablet_manager.cc:1120] Deleting tablet 3c7f072f55a94cca938aa063637af9a5
0507 08:12:20.555198 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.568756 (+ 13558us) spinlock_profiling.cc:243] Waited 12.9 ms on lock 0x6160001d2054. stack: 00007fcb0a83a756 00007fcb0a83a510 00007fcb0986d48d 0000000000955804 00000000009557cd 000000000095519a 00007fcb160a8a00 00007fcb17382b83 00007fcb19e164ec 00007fcb19e48844 00007fcb19e29b30 00007fcb19e298bd 0000000000a60bcc 00007fcb0a880b74 00007fcb0a88419b 00007fcb0a884018
0507 08:12:20.582437 (+ 13681us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.585525 (+ 3088us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.613777 (+ 28252us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":29611392,"tablet-delete.queue_time_us":2861857}
I20240507 08:12:20.653846 5841 tablet_replica.cc:330] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.654037 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3c7f072f55a94cca938aa063637af9a5 (table default.Table_a778d8595fc7496bace57c63194455fe [id=525d572835834c26911e204eb69e8d90]) successfully deleted
I20240507 08:12:20.654078 5841 raft_consensus.cc:2229] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.654233 5841 raft_consensus.cc:2258] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.654778 5843 raft_consensus.cc:348] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.655113 5843 raft_consensus.cc:374] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.655244 5843 raft_consensus.cc:729] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.655607 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2071d1f910a644788fb77006559bb1eb" } new_table_name: "default.Table_3076d02d7ffb4704b6a793f29705bdeb" modify_external_catalogs: true
I20240507 08:12:20.655643 5843 consensus_queue.cc:260] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.654778 5841 ts_tablet_manager.cc:1875] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.658202 7762 raft_consensus.cc:504] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.659080 5843 ts_tablet_manager.cc:1398] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240507 08:12:20.659351 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605be6967a041388dfe81053fbb602e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.659893 5378 leader_election.cc:341] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.660444 5843 tablet_bootstrap.cc:492] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.659689 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ace11ddc318c4b36b5fdb9db738fc029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.661965 5843 tablet_bootstrap.cc:654] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:20.662330 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace11ddc318c4b36b5fdb9db738fc029 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.662606 5839 tablet_bootstrap.cc:492] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.662768 5839 ts_tablet_manager.cc:1367] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20240507 08:12:20.663522 5839 raft_consensus.cc:348] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.663761 5839 raft_consensus.cc:374] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.663856 5839 raft_consensus.cc:729] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.664093 5839 consensus_queue.cc:260] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.664728 5839 ts_tablet_manager.cc:1398] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:20.665237 5839 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.665797 5264 master_service.cc:946] ReplaceTablet: received request to replace tablet c2a3151ab3214a1981f378903bb86a2d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:20.666621 5839 tablet_bootstrap.cc:654] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.669265 5840 tablet_bootstrap.cc:654] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.673126 7743 consensus_queue.cc:1035] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:20.676865 7762 leader_election.cc:288] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:20.677146 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a605be6967a041388dfe81053fbb602e" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.677361 5717 raft_consensus.cc:3037] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.677664 5841 ts_tablet_manager.cc:1888] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.677784 5841 log.cc:1196] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/aca8c38841db45e38ae5d8d965a16d0e
I20240507 08:12:20.678123 5841 ts_tablet_manager.cc:1909] T aca8c38841db45e38ae5d8d965a16d0e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.680462 5717 raft_consensus.cc:2454] T a605be6967a041388dfe81053fbb602e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.680850 5377 leader_election.cc:302] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.681331 7762 raft_consensus.cc:2781] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.681501 7762 raft_consensus.cc:686] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.681875 7762 consensus_queue.cc:237] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.683727 5839 tablet_bootstrap.cc:492] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.683887 5839 ts_tablet_manager.cc:1367] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20240507 08:12:20.684573 5839 raft_consensus.cc:348] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.684667 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1281) took 2991 ms. Trace:
I20240507 08:12:20.684809 5839 raft_consensus.cc:374] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.684758 5841 rpcz_store.cc:276] 0507 08:12:17.693293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.693414 (+ 121us) service_pool.cc:225] Handling call
0507 08:12:20.653775 (+2960361us) ts_tablet_manager.cc:1120] Deleting tablet aca8c38841db45e38ae5d8d965a16d0e
0507 08:12:20.653784 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.668081 (+ 14297us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.677540 (+ 9459us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.684632 (+ 7092us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":16323,"tablet-delete.queue_time_us":2960145}
I20240507 08:12:20.685214 5841 tablet_replica.cc:330] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.685283 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet aca8c38841db45e38ae5d8d965a16d0e (table default.Table_105a7078d5c648b7ac16943b060dbcca [id=dfec8f0e1b45400fb659856ef9d2623c]) successfully deleted
I20240507 08:12:20.684965 5839 raft_consensus.cc:729] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.685798 5839 consensus_queue.cc:260] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.686271 7649 raft_consensus.cc:482] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.686383 7649 raft_consensus.cc:504] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.686928 7649 leader_election.cc:288] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.686662 5843 tablet_bootstrap.cc:492] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.687160 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.687547 5654 leader_election.cc:341] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.687772 5839 ts_tablet_manager.cc:1398] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:20.687544 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.688053 5445 raft_consensus.cc:2454] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:20.688547 5656 leader_election.cc:302] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.688913 7649 raft_consensus.cc:2781] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.688977 7649 raft_consensus.cc:482] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.689024 7649 raft_consensus.cc:3037] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.691744 7649 raft_consensus.cc:504] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.692991 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:20.693153 5445 raft_consensus.cc:3037] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.695843 5445 raft_consensus.cc:2454] T 5ff69b3775d94e5da1f9f327c759abb8 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:20.697044 5656 leader_election.cc:302] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:20.697345 7743 raft_consensus.cc:2781] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.697481 7743 raft_consensus.cc:686] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:20.697805 7743 consensus_queue.cc:237] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.698576 5255 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_85ac61888c9941a195772caf2024259f" } new_table_name: "default.Table_c795e3e1dcbe4533b864459dda2a2ecc" modify_external_catalogs: true
I20240507 08:12:20.699465 7649 leader_election.cc:288] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.687422 5843 ts_tablet_manager.cc:1367] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20240507 08:12:20.703101 5843 raft_consensus.cc:348] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.692644 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ff69b3775d94e5da1f9f327c759abb8" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.703768 5654 leader_election.cc:341] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.703647 5839 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.706280 5843 raft_consensus.cc:374] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.706362 5843 raft_consensus.cc:729] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.707183 5839 tablet_bootstrap.cc:654] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.708283 5843 consensus_queue.cc:260] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:20.708604 5248 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet de7cddb2c4f743f7ae02fd92061a418c (table default.Table_0349bf7e0f9e4be394993e0f579a50b9 [id=f7f693983a8d42f487789d90cebaa872], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet de7cddb2c4f743f7ae02fd92061a418c consensus state has no leader
W20240507 08:12:20.712730 5255 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41ab5762b4704f689744069308c8cb11 has no consensus state
I20240507 08:12:20.713531 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c107b96c8124426cace9b9a1a146589d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.714154 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c107b96c8124426cace9b9a1a146589d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.715130 5843 ts_tablet_manager.cc:1398] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240507 08:12:20.715411 5843 tablet_bootstrap.cc:492] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.717398 5843 tablet_bootstrap.cc:654] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.697790 7079 ts_tablet_manager.cc:1888] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.718640 7079 log.cc:1196] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/b27cc38d0c4c4083900cd6765a5cf6e6
I20240507 08:12:20.719023 7079 ts_tablet_manager.cc:1909] T b27cc38d0c4c4083900cd6765a5cf6e6 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:20.719357 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1303) took 1865 ms. Trace:
I20240507 08:12:20.719446 7079 rpcz_store.cc:276] 0507 08:12:18.854299 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.864951 (+ 10652us) service_pool.cc:225] Handling call
0507 08:12:20.511762 (+1646811us) ts_tablet_manager.cc:1120] Deleting tablet b27cc38d0c4c4083900cd6765a5cf6e6
0507 08:12:20.511770 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.616378 (+104608us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.697679 (+ 81301us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.719339 (+ 21660us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1646572}
I20240507 08:12:20.719925 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet b27cc38d0c4c4083900cd6765a5cf6e6 (table default.Table_1c799c3ed21d4c66bef27973c9e4688d [id=39a45d86f8274b249fe3b4d89ea2ddaf]) successfully deleted
I20240507 08:12:20.721076 7079 tablet_replica.cc:330] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:20.722299 7079 raft_consensus.cc:2229] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.723670 5841 raft_consensus.cc:2229] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.723951 5841 pending_rounds.cc:62] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:20.724016 5841 pending_rounds.cc:68] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924833982414848 op_type: NO_OP noop_request { }
I20240507 08:12:20.724128 5841 raft_consensus.cc:2866] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:20.724223 5841 raft_consensus.cc:2258] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:20.728070 5656 consensus_peers.cc:485] T 4e37b73134cf42ad84b6eec90c9b805f P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.728341 7079 raft_consensus.cc:2258] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.728719 5839 tablet_bootstrap.cc:492] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.728861 5839 ts_tablet_manager.cc:1367] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20240507 08:12:20.728998 7079 ts_tablet_manager.cc:1875] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.729826 5839 raft_consensus.cc:348] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.730155 5839 raft_consensus.cc:374] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.730221 5839 raft_consensus.cc:729] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.730403 5839 consensus_queue.cc:260] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.730971 5839 ts_tablet_manager.cc:1398] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:20.731266 5839 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.731076 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_871e9a698bf647929a1db0c34970c457" } new_table_name: "default.Table_e026d41a44174419bfcc4a533d8faf3c" modify_external_catalogs: true
I20240507 08:12:20.733131 5841 ts_tablet_manager.cc:1875] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.733856 6513 ts_tablet_manager.cc:1888] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.733944 6513 log.cc:1196] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/99e4514039304f0696c69a057db0ea6e
I20240507 08:12:20.734279 6513 ts_tablet_manager.cc:1909] T 99e4514039304f0696c69a057db0ea6e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.735342 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_02a149a05852404a912a65ef0c6bc205" } modify_external_catalogs: true
I20240507 08:12:20.735499 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_02a149a05852404a912a65ef0c6bc205" } modify_external_catalogs: true
I20240507 08:12:20.739600 5841 ts_tablet_manager.cc:1888] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.739756 5841 log.cc:1196] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/4e37b73134cf42ad84b6eec90c9b805f
I20240507 08:12:20.740206 5841 ts_tablet_manager.cc:1909] T 4e37b73134cf42ad84b6eec90c9b805f P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.740562 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1279) took 3047 ms. Trace:
I20240507 08:12:20.740798 5841 rpcz_store.cc:276] 0507 08:12:17.693070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.693341 (+ 271us) service_pool.cc:225] Handling call
0507 08:12:20.685091 (+2991750us) ts_tablet_manager.cc:1120] Deleting tablet 4e37b73134cf42ad84b6eec90c9b805f
0507 08:12:20.685101 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.723857 (+ 38756us) spinlock_profiling.cc:243] Waited 38.2 ms on lock 0x6160000f4054. stack: 00007fcb0a83a756 00007fcb0a83a510 00007fcb0986d48d 0000000000955804 00000000009557cd 000000000095519a 00007fcb160a8a00 00007fcb17382b83 00007fcb19e164ec 00007fcb19e48844 00007fcb19e29b30 00007fcb19e298bd 0000000000a60bcc 00007fcb0a880b74 00007fcb0a88419b 00007fcb0a884018
0507 08:12:20.736846 (+ 12989us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.739514 (+ 2668us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.740532 (+ 1018us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":110,"spinlock_wait_cycles":87942400,"tablet-delete.queue_time_us":2975210}
I20240507 08:12:20.741398 5841 tablet_replica.cc:330] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.741472 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 4e37b73134cf42ad84b6eec90c9b805f (table default.Table_dd26080257a24a50932dc596426fc19a [id=e917ed6f3e4a47bb85c0799b4d7d0f71]) successfully deleted
I20240507 08:12:20.741665 5841 raft_consensus.cc:2229] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.742033 7720 raft_consensus.cc:482] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.742153 7720 raft_consensus.cc:504] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.742544 5839 tablet_bootstrap.cc:654] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.742771 7720 leader_election.cc:288] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:20.743471 5520 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: f2de26190bc44916809856d84ad68345
W20240507 08:12:20.743780 5517 leader_election.cc:341] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: f2de26190bc44916809856d84ad68345
I20240507 08:12:20.743877 5517 leader_election.cc:302] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.744202 7720 raft_consensus.cc:2726] T f2de26190bc44916809856d84ad68345 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.745260 5843 tablet_bootstrap.cc:492] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.745429 5843 ts_tablet_manager.cc:1367] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.001s
I20240507 08:12:20.746080 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1237) took 4575 ms. Trace:
W20240507 08:12:20.746156 5656 consensus_peers.cc:485] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 08aac6e88b7c41d8a59d1e12e7f6c14e. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.746335 5843 raft_consensus.cc:348] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.746570 5843 raft_consensus.cc:374] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.746651 5843 raft_consensus.cc:729] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.746845 5843 consensus_queue.cc:260] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.747416 5843 ts_tablet_manager.cc:1398] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:20.748143 5843 tablet_bootstrap.cc:492] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.750165 5257 master_service.cc:946] ReplaceTablet: received request to replace tablet b15c8ad2a8324c75a1ef49e219b962cf from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:20.754050 5843 tablet_bootstrap.cc:654] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.756927 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_858753a9d5084a709cac3cf93887cf25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.757452 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_858753a9d5084a709cac3cf93887cf25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.758262 5839 tablet_bootstrap.cc:492] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.746170 6513 rpcz_store.cc:276] 0507 08:12:16.170431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.170511 (+ 80us) service_pool.cc:225] Handling call
0507 08:12:20.637221 (+4466710us) ts_tablet_manager.cc:1120] Deleting tablet 99e4514039304f0696c69a057db0ea6e
0507 08:12:20.637248 (+ 27us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.680273 (+ 43025us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.733760 (+ 53487us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.746062 (+ 12302us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4466543}
I20240507 08:12:20.758417 5839 ts_tablet_manager.cc:1367] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20240507 08:12:20.758620 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 99e4514039304f0696c69a057db0ea6e (table default.Table_fc64369fe72742ccb59a630ee95a8007 [id=54ceed7b5cad41ecbacc40ace9f48cd5]) successfully deleted
I20240507 08:12:20.759094 6513 tablet_replica.cc:330] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.759285 6513 raft_consensus.cc:2229] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:20.759261 5839 raft_consensus.cc:348] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.759526 5839 raft_consensus.cc:374] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.759644 5839 raft_consensus.cc:729] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.759873 5839 consensus_queue.cc:260] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.760361 6513 raft_consensus.cc:2258] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.760547 5839 ts_tablet_manager.cc:1398] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:20.760834 5839 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.762257 6513 ts_tablet_manager.cc:1875] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.769714 5839 tablet_bootstrap.cc:654] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.771186 5843 tablet_bootstrap.cc:492] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.771345 5843 ts_tablet_manager.cc:1367] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240507 08:12:20.771643 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2cd819bfa63841308bf0554cd623e25a" } new_table_name: "default.Table_fc4bab99801a401b836084c924d7cafd" modify_external_catalogs: true
I20240507 08:12:20.772079 5843 raft_consensus.cc:348] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.772333 5843 raft_consensus.cc:374] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.772418 5843 raft_consensus.cc:729] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.772676 5843 consensus_queue.cc:260] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.773747 5840 tablet_bootstrap.cc:492] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:20.773914 5840 ts_tablet_manager.cc:1367] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.127s user 0.003s sys 0.003s
I20240507 08:12:20.774580 5840 raft_consensus.cc:348] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.774840 5840 raft_consensus.cc:374] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.774907 5840 raft_consensus.cc:729] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.775194 5840 consensus_queue.cc:260] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.777029 5840 ts_tablet_manager.cc:1398] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240507 08:12:20.777319 5843 ts_tablet_manager.cc:1398] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20240507 08:12:20.777441 5840 tablet_bootstrap.cc:492] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:20.778278 5843 tablet_bootstrap.cc:492] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:20.779595 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c59886326a48434fb737976d9f571cd3 consensus state has no leader
I20240507 08:12:20.780371 7736 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:20.781262 7732 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:20.781675 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_dac9feb69fbd4fab970744ba42ac0d51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.782582 5843 tablet_bootstrap.cc:654] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:20.784515 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dac9feb69fbd4fab970744ba42ac0d51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.785131 5839 tablet_bootstrap.cc:492] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.785282 5839 ts_tablet_manager.cc:1367] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.002s
I20240507 08:12:20.786136 5839 raft_consensus.cc:348] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.786372 5839 raft_consensus.cc:374] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.786434 5839 raft_consensus.cc:729] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.786906 5839 consensus_queue.cc:260] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.787533 5839 ts_tablet_manager.cc:1398] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:20.788946 5839 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.790067 7079 ts_tablet_manager.cc:1888] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.794072 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0e463b9da334467b9a3a86030bcb202d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.795078 7762 raft_consensus.cc:482] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.795238 7762 raft_consensus.cc:504] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.795348 7732 tablet.cc:1722] T b7b4eb72b855459b8da46c037108097e P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240507 08:12:20.796168 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.796530 7079 log.cc:1196] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/957775c01be94cc081db344c95a81310
I20240507 08:12:20.796841 7079 ts_tablet_manager.cc:1909] T 957775c01be94cc081db344c95a81310 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:20.796803 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.797169 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1305) took 1924 ms. Trace:
I20240507 08:12:20.797235 7079 rpcz_store.cc:276] 0507 08:12:18.872518 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.872580 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:20.721028 (+1848448us) ts_tablet_manager.cc:1120] Deleting tablet 957775c01be94cc081db344c95a81310
0507 08:12:20.721038 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.747118 (+ 26080us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.789968 (+ 42850us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.797154 (+ 7186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1846669}
I20240507 08:12:20.797529 7079 tablet_replica.cc:330] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:20.797466 5378 leader_election.cc:341] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.797703 7079 raft_consensus.cc:2229] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.797852 7079 raft_consensus.cc:2258] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:20.798743 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e463b9da334467b9a3a86030bcb202d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.801720 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 957775c01be94cc081db344c95a81310 (table default.Table_d3d5d2e042244132918d76affc48a306 [id=46a4d7cec48d4fbebd8d895d6ef640b2]) successfully deleted
W20240507 08:12:20.802067 5377 leader_election.cc:341] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.802179 5377 leader_election.cc:302] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.802435 7722 raft_consensus.cc:2726] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.802930 7736 tablet.cc:1722] T b7b4eb72b855459b8da46c037108097e P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240507 08:12:20.807559 5656 consensus_peers.cc:485] T e7eef600626a4961ab1fae27162537bb P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.809980 7750 tablet.cc:1722] T e4c064b969844d938fddfe7f043c0b92 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20240507 08:12:20.816097 7720 raft_consensus.cc:482] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.816211 7720 raft_consensus.cc:504] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.816751 7720 leader_election.cc:288] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.818656 7762 leader_election.cc:288] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:12:20.819394 5520 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b1fadbfd330b4e699e151f83125da281
W20240507 08:12:20.821661 5517 leader_election.cc:341] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: b1fadbfd330b4e699e151f83125da281
I20240507 08:12:20.821756 5517 leader_election.cc:302] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.822144 7720 raft_consensus.cc:2726] T b1fadbfd330b4e699e151f83125da281 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.790292 5839 tablet_bootstrap.cc:654] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.830659 7720 raft_consensus.cc:482] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.830770 7720 raft_consensus.cc:504] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.831070 7079 ts_tablet_manager.cc:1875] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.831332 7720 leader_election.cc:288] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:20.831864 5656 consensus_peers.cc:485] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e17502bb4ac64f6ca9c77a3b00ab9f4a. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.832171 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7c43f4862b6b405eab680fc1856712a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.832300 5517 leader_election.cc:341] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 055a30bfc61b4ff2be9f2418ca59b030
I20240507 08:12:20.832798 5839 tablet_bootstrap.cc:492] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
W20240507 08:12:20.832937 5520 leader_election.cc:341] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 055a30bfc61b4ff2be9f2418ca59b030
I20240507 08:12:20.833019 5520 leader_election.cc:302] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.833320 7720 raft_consensus.cc:2726] T 055a30bfc61b4ff2be9f2418ca59b030 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.833593 5839 ts_tablet_manager.cc:1367] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20240507 08:12:20.834352 7748 raft_consensus.cc:482] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.834522 7748 raft_consensus.cc:504] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.834565 5839 raft_consensus.cc:348] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.834725 5839 raft_consensus.cc:374] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.834782 5839 raft_consensus.cc:729] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.835263 7748 leader_election.cc:288] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.835729 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.836211 5654 leader_election.cc:341] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.836627 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
W20240507 08:12:20.836683 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c43f4862b6b405eab680fc1856712a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:20.837096 5656 leader_election.cc:341] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.837199 5656 leader_election.cc:302] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4; no voters: dc665657d5b34fd7aec884e0f2668e11, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.837417 7748 raft_consensus.cc:2726] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.837853 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1cbd9a6d4b484b8a9f7c722a8bdde027
I20240507 08:12:20.837898 5839 consensus_queue.cc:260] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.838558 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7bbfcef77950434d9b09839fb0bac453" } new_table_name: "default.Table_0fd0c098eb9b49478377ce137b555bf9" modify_external_catalogs: true
I20240507 08:12:20.840325 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.840564 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.841322 5839 ts_tablet_manager.cc:1398] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240507 08:12:20.841560 5839 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.841811 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.842578 5840 tablet_bootstrap.cc:654] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:20.847944 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c6d972e15fa840668e74ac1416c2e2d9 consensus state has no leader
I20240507 08:12:20.849108 5581 raft_consensus.cc:3037] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.849884 5839 tablet_bootstrap.cc:654] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.851334 5841 raft_consensus.cc:2258] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.851884 5841 ts_tablet_manager.cc:1875] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.853471 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a45c704ec20a4128af7446f00f9fc5af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.854059 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a45c704ec20a4128af7446f00f9fc5af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.854323 5843 tablet_bootstrap.cc:492] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.854494 5843 ts_tablet_manager.cc:1367] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.076s user 0.007s sys 0.000s
I20240507 08:12:20.855204 5843 raft_consensus.cc:348] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.855423 5843 raft_consensus.cc:374] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.855525 5843 raft_consensus.cc:729] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.855755 5843 consensus_queue.cc:260] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.856395 5843 ts_tablet_manager.cc:1398] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:20.856686 5843 tablet_bootstrap.cc:492] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.859450 5841 ts_tablet_manager.cc:1888] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.860368 5841 log.cc:1196] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/e7eef600626a4961ab1fae27162537bb
I20240507 08:12:20.863127 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 35678404b67b4273b61fa5f922152fe3
I20240507 08:12:20.863837 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_30f4473a00de4322aacef85c316e67f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.864357 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30f4473a00de4322aacef85c316e67f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.864585 7762 raft_consensus.cc:482] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.864692 5843 tablet_bootstrap.cc:654] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.864706 7762 raft_consensus.cc:504] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.865322 7762 leader_election.cc:288] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.865453 5841 ts_tablet_manager.cc:1909] T e7eef600626a4961ab1fae27162537bb P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.866060 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31fdadee60e4f5e94e92c536e18774d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:20.866384 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1282) took 3146 ms. Trace:
I20240507 08:12:20.866470 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:20.866652 5378 leader_election.cc:341] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.867308 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.868098 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.868131 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31fdadee60e4f5e94e92c536e18774d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.868317 5717 raft_consensus.cc:2454] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.868631 5377 leader_election.cc:302] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.869055 7762 raft_consensus.cc:2781] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.869123 7762 raft_consensus.cc:482] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.869176 7762 raft_consensus.cc:3037] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.870632 5839 tablet_bootstrap.cc:492] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.870755 5839 ts_tablet_manager.cc:1367] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20240507 08:12:20.871201 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_3c9ab59af0ed44298ce2cad9c666645a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:20.871388 5839 raft_consensus.cc:348] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.871598 5839 raft_consensus.cc:374] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.871651 5839 raft_consensus.cc:729] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:20.871650 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c9ab59af0ed44298ce2cad9c666645a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.871631 7762 raft_consensus.cc:504] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.871845 5839 consensus_queue.cc:260] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.872400 5839 ts_tablet_manager.cc:1398] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:20.872654 5839 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.872924 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31fdadee60e4f5e94e92c536e18774d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.873096 5717 raft_consensus.cc:3037] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.873189 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b31fdadee60e4f5e94e92c536e18774d" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.873878 5378 leader_election.cc:341] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.874249 7762 leader_election.cc:288] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.874998 5839 tablet_bootstrap.cc:654] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.878616 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_182f131640294c91ab16e34d51ee14ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.879031 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_182f131640294c91ab16e34d51ee14ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.880453 5717 raft_consensus.cc:2454] T b31fdadee60e4f5e94e92c536e18774d P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.880793 5377 leader_election.cc:302] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.881175 7762 raft_consensus.cc:2781] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.881399 7762 raft_consensus.cc:686] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.881680 7762 consensus_queue.cc:237] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.883508 7722 raft_consensus.cc:482] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.883605 7722 raft_consensus.cc:504] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.866458 5841 rpcz_store.cc:276] 0507 08:12:17.720316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.720389 (+ 73us) service_pool.cc:225] Handling call
0507 08:12:20.741327 (+3020938us) ts_tablet_manager.cc:1120] Deleting tablet e7eef600626a4961ab1fae27162537bb
0507 08:12:20.741336 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.854784 (+113448us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.859354 (+ 4570us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.866366 (+ 7012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3020762}
I20240507 08:12:20.883987 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet e7eef600626a4961ab1fae27162537bb (table default.Table_73451659761f47548c957eac0689a2f5 [id=e76965f068da4f158275fbed933694ce]) successfully deleted
I20240507 08:12:20.884106 7722 leader_election.cc:288] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.884550 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:20.885093 5378 leader_election.cc:341] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.885936 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.886796 7720 raft_consensus.cc:482] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.886806 5717 raft_consensus.cc:2454] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:20.886921 7720 raft_consensus.cc:504] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.887405 7720 leader_election.cc:288] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.887758 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_acc44d6917804c998a92bdc6e7799e2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.888221 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc44d6917804c998a92bdc6e7799e2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:20.888293 5520 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:12:20.889070 5377 leader_election.cc:302] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.889463 7722 raft_consensus.cc:2781] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:20.889533 7722 raft_consensus.cc:482] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:20.889608 7722 raft_consensus.cc:3037] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.890904 5841 tablet_replica.cc:330] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.891100 5841 raft_consensus.cc:2229] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.891248 5841 raft_consensus.cc:2258] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:20.892886 5517 leader_election.cc:341] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 84212d6cb2c742c1a0f3fdae8be8ebe1
I20240507 08:12:20.893010 5517 leader_election.cc:302] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.894551 7722 raft_consensus.cc:504] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.894905 7720 raft_consensus.cc:2726] T 84212d6cb2c742c1a0f3fdae8be8ebe1 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:20.895975 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:20.896185 5717 raft_consensus.cc:3037] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:20.896513 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "093fb3986a674b93a435408c7ec85500" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:20.897624 5378 leader_election.cc:341] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:20.897841 7722 leader_election.cc:288] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:20.900589 5841 ts_tablet_manager.cc:1875] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.902339 5259 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 38ce7a166001486197b78f8b02a437ed
I20240507 08:12:20.903740 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ce7a166001486197b78f8b02a437ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.905920 5263 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 372dd1ea1a4043c9ad5afb1d48dfc133
I20240507 08:12:20.906903 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ce7a166001486197b78f8b02a437ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.906994 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:20.907088 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.903236 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ce7a166001486197b78f8b02a437ed with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.906699 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:20.914870 5717 raft_consensus.cc:2454] T 093fb3986a674b93a435408c7ec85500 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:20.915617 5839 tablet_bootstrap.cc:492] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.915753 5839 ts_tablet_manager.cc:1367] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.043s user 0.015s sys 0.002s
I20240507 08:12:20.916159 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d036e0ee03b344c1872fd1624fe92142" } new_table_name: "default.Table_25ea38c9985e402db419e238f23f9f36" modify_external_catalogs: true
I20240507 08:12:20.916438 5839 raft_consensus.cc:348] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.916528 5377 leader_election.cc:302] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:20.916657 5839 raft_consensus.cc:374] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.916712 5839 raft_consensus.cc:729] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.916929 7762 raft_consensus.cc:2781] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:20.916898 5839 consensus_queue.cc:260] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.917309 7762 raft_consensus.cc:686] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:20.917415 5839 ts_tablet_manager.cc:1398] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:20.917678 5839 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.917682 7762 consensus_queue.cc:237] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.919200 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6804b703b4a240d389ddf5b42892ed4e" } new_table_name: "default.Table_baf4dd7a9fef47a08fbd55804100dba3" modify_external_catalogs: true
I20240507 08:12:20.919631 5839 tablet_bootstrap.cc:654] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.921864 7720 raft_consensus.cc:482] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:20.921970 7720 raft_consensus.cc:504] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.922540 7720 leader_election.cc:288] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:20.922806 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:20.923084 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "645ae26bb26f43d298afe98817d88fa4" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:20.923669 5520 leader_election.cc:302] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:20.924111 7720 raft_consensus.cc:2726] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:20.926259 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c053869c840e4773a3c8e07a8b426b28 consensus state has no leader
I20240507 08:12:20.929061 5841 ts_tablet_manager.cc:1888] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.930105 5841 log.cc:1196] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3fce98f472ef46589dbea6b6aefa6572
W20240507 08:12:20.930749 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d0d15ec7033c4647b10e4085a097eebb has no consensus state
I20240507 08:12:20.931771 5841 ts_tablet_manager.cc:1909] T 3fce98f472ef46589dbea6b6aefa6572 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.933037 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1291) took 3050 ms. Trace:
I20240507 08:12:20.933634 5839 tablet_bootstrap.cc:492] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.933797 5839 ts_tablet_manager.cc:1367] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.006s
I20240507 08:12:20.934049 5841 rpcz_store.cc:276] 0507 08:12:17.882907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.883066 (+ 159us) service_pool.cc:225] Handling call
0507 08:12:20.890858 (+3007792us) ts_tablet_manager.cc:1120] Deleting tablet 3fce98f472ef46589dbea6b6aefa6572
0507 08:12:20.890868 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.910854 (+ 19986us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.928967 (+ 18113us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.933020 (+ 4053us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3007614}
I20240507 08:12:20.934712 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3fce98f472ef46589dbea6b6aefa6572 (table default.Table_727fd7e435fd40368f63b993db76af91 [id=786a00558b0f42cd9700cd3200dfe1ac]) successfully deleted
I20240507 08:12:20.934638 5839 raft_consensus.cc:348] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.934809 5839 raft_consensus.cc:374] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.934859 5839 raft_consensus.cc:729] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.935119 5841 tablet_replica.cc:330] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.935074 5839 consensus_queue.cc:260] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:20.935573 5841 raft_consensus.cc:2229] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.935601 5839 ts_tablet_manager.cc:1398] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:20.935876 5839 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.936050 5841 raft_consensus.cc:2258] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.936789 5841 ts_tablet_manager.cc:1875] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.937564 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_b730eec0bfab44919c1e1a27ca44701c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:20.938047 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b730eec0bfab44919c1e1a27ca44701c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.938088 6513 ts_tablet_manager.cc:1888] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:20.938426 6513 log.cc:1196] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/8bb62a72ed2248f6886823b9c64ff4c6
I20240507 08:12:20.938762 6513 ts_tablet_manager.cc:1909] T 8bb62a72ed2248f6886823b9c64ff4c6 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:20.939061 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1239) took 4621 ms. Trace:
I20240507 08:12:20.939148 6513 rpcz_store.cc:276] 0507 08:12:16.317399 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.317455 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:20.759048 (+4441593us) ts_tablet_manager.cc:1120] Deleting tablet 8bb62a72ed2248f6886823b9c64ff4c6
0507 08:12:20.759058 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.805344 (+ 46286us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.937985 (+132641us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.939040 (+ 1055us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4441351}
I20240507 08:12:20.939337 5839 tablet_bootstrap.cc:654] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.939493 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 8bb62a72ed2248f6886823b9c64ff4c6 (table default.Table_f9c3b5637d424b1199065768db5c961a [id=866842c12bfe4b5aa14e08e8e3e226e3]) successfully deleted
I20240507 08:12:20.939646 6513 tablet_replica.cc:330] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:20.939973 6513 raft_consensus.cc:2229] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.940233 6513 raft_consensus.cc:2258] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:20.941120 6513 ts_tablet_manager.cc:1875] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:20.945240 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_9bf0c5211ab143cf877ed2d2fbf8c2b5" } new_table_name: "default.Table_1e08f922a69a40a4b1c59dc5f3e83488" modify_external_catalogs: true
W20240507 08:12:20.953466 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f9d270dcfe84128a31fe48d5f2be090 consensus state has no leader
I20240507 08:12:20.954452 5839 tablet_bootstrap.cc:492] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.954573 5839 ts_tablet_manager.cc:1367] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20240507 08:12:20.955276 5839 raft_consensus.cc:348] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.955451 5839 raft_consensus.cc:374] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.955507 5839 raft_consensus.cc:729] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.955739 5839 consensus_queue.cc:260] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.956419 5839 ts_tablet_manager.cc:1398] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:20.956717 5839 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.959767 5839 tablet_bootstrap.cc:654] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.964936 5841 ts_tablet_manager.cc:1888] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:20.965790 5841 log.cc:1196] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3ccc3e67eeb34c42b36edde7bacd23a5
I20240507 08:12:20.966267 5841 ts_tablet_manager.cc:1909] T 3ccc3e67eeb34c42b36edde7bacd23a5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:20.967079 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet a605be6967a041388dfe81053fbb602e (table default.Table_13bfcb3083e54a37b5fad286ae0e5b26 [id=b92037dbac744552b58099b7fdc424b2]). Expected version 1 got 0
I20240507 08:12:20.967564 5261 catalog_manager.cc:5484] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:20.967953 5261 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet 66247e08dfdf47f69ba24b074c8080f0 (table default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7]). Expected version 2 got 0
W20240507 08:12:20.977056 5088 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 76078.7us
I20240507 08:12:20.978516 5839 tablet_bootstrap.cc:492] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.978643 5839 ts_tablet_manager.cc:1367] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20240507 08:12:20.979422 5839 raft_consensus.cc:348] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.979588 5839 raft_consensus.cc:374] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.979640 5839 raft_consensus.cc:729] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
W20240507 08:12:20.981783 5654 consensus_peers.cc:485] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:20.981974 7750 tablet.cc:1722] T bcefbe3a60f84cadb5fd3e69c6efdd02 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:20.982009 5839 consensus_queue.cc:260] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.982503 5839 ts_tablet_manager.cc:1398] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240507 08:12:20.982729 5839 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.983914 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1290) took 3101 ms. Trace:
I20240507 08:12:20.984045 5841 rpcz_store.cc:276] 0507 08:12:17.882789 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.883759 (+ 970us) service_pool.cc:225] Handling call
0507 08:12:20.935077 (+3051318us) ts_tablet_manager.cc:1120] Deleting tablet 3ccc3e67eeb34c42b36edde7bacd23a5
0507 08:12:20.935085 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.948725 (+ 13640us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.964802 (+ 16077us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:20.983892 (+ 19090us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3046800}
I20240507 08:12:20.984455 5839 tablet_bootstrap.cc:654] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:20.973927 5843 tablet_bootstrap.cc:492] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:20.985014 5843 ts_tablet_manager.cc:1367] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.128s user 0.019s sys 0.002s
I20240507 08:12:20.985868 5843 raft_consensus.cc:348] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.986127 5843 raft_consensus.cc:374] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.986186 5843 raft_consensus.cc:729] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.986407 5843 consensus_queue.cc:260] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.987051 5843 ts_tablet_manager.cc:1398] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:20.987327 5843 tablet_bootstrap.cc:492] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.991036 5839 tablet_bootstrap.cc:492] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:20.991151 5839 ts_tablet_manager.cc:1367] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240507 08:12:20.991755 5839 raft_consensus.cc:348] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.991954 5839 raft_consensus.cc:374] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:20.991997 5839 raft_consensus.cc:729] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.992226 5839 consensus_queue.cc:260] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.992686 5839 ts_tablet_manager.cc:1398] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240507 08:12:20.992920 5839 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:20.995234 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet c839c8a5041f4c3da9bc0749afee1413 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.995455 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet 63ccdad494794192b3c37c8d5cbfcb17 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:20.984750 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1297) took 3026 ms. Trace:
I20240507 08:12:20.995864 5841 rpcz_store.cc:276] 0507 08:12:17.958117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.958171 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:20.984615 (+3026444us) ts_tablet_manager.cc:1120] Deleting tablet d718aa8fe99041688964d1a05cef49a0
0507 08:12:20.984624 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.984732 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3026150}
I20240507 08:12:20.996351 5841 tablet_replica.cc:330] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:20.996569 5841 raft_consensus.cc:2229] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:20.996940 5841 raft_consensus.cc:2258] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.001549 5839 tablet_bootstrap.cc:654] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:21.008116 5654 consensus_peers.cc:485] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.008458 5840 tablet_bootstrap.cc:492] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.008565 5840 ts_tablet_manager.cc:1367] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.231s user 0.003s sys 0.000s
I20240507 08:12:21.009200 5840 raft_consensus.cc:348] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.009369 5840 raft_consensus.cc:374] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.009410 5840 raft_consensus.cc:729] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:20.970136 5262 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_c7e3733ff21f4f19b964fae8b1150d8d [id=06457cc0f83a470a95273688e948d21a] alter complete (version 2)
I20240507 08:12:21.009630 5840 consensus_queue.cc:260] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:20.973520 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_66582e1db1394753bb2b86a9ece6d785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.010752 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66582e1db1394753bb2b86a9ece6d785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.984493 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3ccc3e67eeb34c42b36edde7bacd23a5 (table default.Table_73b2cacc437f4f8791c29d803ea49436 [id=b591a06a8e02403ab74223596a6f8acb]) successfully deleted
W20240507 08:12:21.013480 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d718aa8fe99041688964d1a05cef49a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d718aa8fe99041688964d1a05cef49a0 already in progress: creating tablet
I20240507 08:12:20.983179 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_a894109d6af4424db5114275d258a4f0" } new_table_name: "default.Table_a38f7954549c46af8849adfa845ea1fd" modify_external_catalogs: true
I20240507 08:12:20.994558 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c1f3dc0a02c948e7ad05ddde8f2d58c1
I20240507 08:12:20.994736 5264 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet c2a3151ab3214a1981f378903bb86a2d
I20240507 08:12:21.016855 5264 catalog_manager.cc:6554] ReplaceTablet: tablet c2a3151ab3214a1981f378903bb86a2d deleted and replaced by tablet 8a24a3bd2c624ecb8a46896c0325db93
I20240507 08:12:20.995810 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_2fd519a4f0904d04a2f490068d57e8a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.017143 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3151ab3214a1981f378903bb86a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a24a3bd2c624ecb8a46896c0325db93) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.017143 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:21.017464 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fd519a4f0904d04a2f490068d57e8a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:20.968596 5320 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 35678404b67b4273b61fa5f922152fe3
I20240507 08:12:21.018229 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.019901 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 04c410e738234de590b7aecd80aebee8 (DEFAULT_TABLE table=default.Table_3df807895f4648f18a0eb83bb6307de0 [id=cccef4ba6d2a4f45a18738e2f49020d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.020186 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.020370 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3151ab3214a1981f378903bb86a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a24a3bd2c624ecb8a46896c0325db93) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.020502 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.009718 7762 raft_consensus.cc:482] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.020808 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c410e738234de590b7aecd80aebee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.013868 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fac82e12d0437e950d01dbc4117efa with delete_type TABLET_DATA_DELETED (replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.020879 7762 raft_consensus.cc:504] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.021499 7762 leader_election.cc:288] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.022754 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec08c0d8a1ab444489340415303a5739" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.023923 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 35678404b67b4273b61fa5f922152fe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.024080 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a3151ab3214a1981f378903bb86a2d with delete_type TABLET_DATA_DELETED (replaced by tablet 8a24a3bd2c624ecb8a46896c0325db93) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.024226 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:20 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.015100 5843 tablet_bootstrap.cc:654] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:21.028183 5378 leader_election.cc:341] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.029327 5841 ts_tablet_manager.cc:1875] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.032099 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5842a3cae56b41ff803ba5aa759a0823"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.032573 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5842a3cae56b41ff803ba5aa759a0823 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.034446 5843 tablet_bootstrap.cc:492] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.034564 5843 ts_tablet_manager.cc:1367] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.002s
I20240507 08:12:21.035239 5843 raft_consensus.cc:348] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.035403 5843 raft_consensus.cc:374] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.035465 5843 raft_consensus.cc:729] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.035658 5843 consensus_queue.cc:260] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.036149 5843 ts_tablet_manager.cc:1398] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:21.036386 5843 tablet_bootstrap.cc:492] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:20.988507 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 0c4cd7b56d4e468eb754d9653117d023 (table default.Table_f7583788c28a4279a8cd5d8684eb4393 [id=aac50aefe6524a84b1e4564d3f0ce394]). Expected version 4 got 0
I20240507 08:12:21.037367 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet bcefbe3a60f84cadb5fd3e69c6efdd02 (table default.Table_e3bcf078654d42ed8331d033b29e482c [id=ca5e7cb9fb9e41c8a4fda41af0c5144e]). Expected version 2 got 0
W20240507 08:12:21.037628 5656 consensus_peers.cc:485] T 47a154769d0f4501837888feba98f33d P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 47a154769d0f4501837888feba98f33d. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.037724 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec08c0d8a1ab444489340415303a5739" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.037865 5843 tablet_bootstrap.cc:654] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.037941 5717 raft_consensus.cc:2454] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:21.038365 5257 catalog_manager.cc:5576] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165) does not have the latest schema for tablet ec08c0d8a1ab444489340415303a5739 (table default.Table_73aa505d7d4a4389bbd269ee956f5d8a [id=c4b899bf5b0244ac897da574bda1457d]). Expected version 1 got 0
I20240507 08:12:21.038636 5257 catalog_manager.cc:5484] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cbb887175f345ada2433adaec906fd4 (127.4.246.195). New cstate: current_term: 1 leader_uuid: "1cbb887175f345ada2433adaec906fd4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: UNKNOWN } } }
I20240507 08:12:21.040444 5377 leader_election.cc:302] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.041482 7762 raft_consensus.cc:2781] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.041656 7762 raft_consensus.cc:482] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.041626 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_388837be9b1b4e38b8033dc631d6ed2f" } modify_external_catalogs: true
I20240507 08:12:21.041744 7762 raft_consensus.cc:3037] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.041803 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_388837be9b1b4e38b8033dc631d6ed2f" } modify_external_catalogs: true
I20240507 08:12:21.042563 5632 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:21.044746 7762 raft_consensus.cc:504] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.045787 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec08c0d8a1ab444489340415303a5739" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:21.045953 5717 raft_consensus.cc:3037] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.046067 7762 leader_election.cc:288] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.047660 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec08c0d8a1ab444489340415303a5739" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.048117 5378 leader_election.cc:341] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.048367 5717 raft_consensus.cc:2454] T ec08c0d8a1ab444489340415303a5739 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:21.049358 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d3a39ab037194d1b9c28f520890fa695" } new_table_name: "default.Table_088f39260703438992510c7ce697ac44" modify_external_catalogs: true
I20240507 08:12:21.050637 5377 leader_election.cc:302] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.051126 7762 raft_consensus.cc:2781] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.051410 7762 raft_consensus.cc:686] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:21.051704 7762 consensus_queue.cc:237] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.056106 5841 ts_tablet_manager.cc:1888] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.059410 7720 raft_consensus.cc:482] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.059540 7720 raft_consensus.cc:504] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.060148 7720 leader_election.cc:288] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
W20240507 08:12:21.060665 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c383c49d5974eee9f19cedb43fa812f consensus state has no leader
W20240507 08:12:21.061396 5520 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 3f01e3311a8b4553a31914922fb08d13
I20240507 08:12:21.063891 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet ec08c0d8a1ab444489340415303a5739 (table default.Table_73aa505d7d4a4389bbd269ee956f5d8a [id=c4b899bf5b0244ac897da574bda1457d]). Expected version 1 got 0
W20240507 08:12:21.064749 5517 leader_election.cc:341] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 3f01e3311a8b4553a31914922fb08d13
I20240507 08:12:21.064853 5517 leader_election.cc:302] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.065236 7720 raft_consensus.cc:2726] T 3f01e3311a8b4553a31914922fb08d13 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.065733 5264 catalog_manager.cc:5484] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:21.067312 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet a605be6967a041388dfe81053fbb602e (table default.Table_13bfcb3083e54a37b5fad286ae0e5b26 [id=b92037dbac744552b58099b7fdc424b2]). Expected version 1 got 0
I20240507 08:12:21.067520 5841 log.cc:1196] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/3a94c97ecc5d47abb190f3833fd317ca
W20240507 08:12:21.069862 5656 consensus_peers.cc:485] T 3a94c97ecc5d47abb190f3833fd317ca P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.070382 5841 ts_tablet_manager.cc:1909] T 3a94c97ecc5d47abb190f3833fd317ca P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.070752 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 01a9165fdc41482cb1f8431ab6479126 (table default.Table_206b231a053b42fab47d3c3973502c90 [id=825a7b456d14442ab083c2700bed6a98]). Expected version 1 got 0
I20240507 08:12:21.071606 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1298) took 3079 ms. Trace:
I20240507 08:12:21.074847 5841 rpcz_store.cc:276] 0507 08:12:17.992555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.992702 (+ 147us) service_pool.cc:225] Handling call
0507 08:12:20.996298 (+3003596us) ts_tablet_manager.cc:1120] Deleting tablet 3a94c97ecc5d47abb190f3833fd317ca
0507 08:12:20.996307 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.032804 (+ 36497us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.056010 (+ 23206us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.071590 (+ 15580us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3003393}
I20240507 08:12:21.075320 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 3a94c97ecc5d47abb190f3833fd317ca (table default.Table_18ef77bde60b44179d9367054bffc7af [id=b40ed3eac5e4406299c3ad2934df75db]) successfully deleted
I20240507 08:12:21.075718 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1303) took 3039 ms. Trace:
I20240507 08:12:21.077468 5840 ts_tablet_manager.cc:1398] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.069s user 0.002s sys 0.000s
I20240507 08:12:21.077780 5840 tablet_bootstrap.cc:492] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.080687 5841 rpcz_store.cc:276] 0507 08:12:18.035950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.036471 (+ 521us) service_pool.cc:225] Handling call
0507 08:12:21.075614 (+3039143us) ts_tablet_manager.cc:1120] Deleting tablet cd4d1fa6a8d94af982e1d23c08cbdeac
0507 08:12:21.075623 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.075704 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3038991}
W20240507 08:12:21.081329 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet cd4d1fa6a8d94af982e1d23c08cbdeac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cd4d1fa6a8d94af982e1d23c08cbdeac already in progress: creating tablet
I20240507 08:12:21.081849 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1304) took 3023 ms. Trace:
I20240507 08:12:21.082137 5840 tablet_bootstrap.cc:654] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.083022 5841 rpcz_store.cc:276] 0507 08:12:18.058593 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.058643 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:21.081702 (+3023059us) ts_tablet_manager.cc:1120] Deleting tablet 8d1a4e05273b4f8c83dc8c7825a39310
0507 08:12:21.081715 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.081830 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3014935}
I20240507 08:12:21.083519 5839 tablet_bootstrap.cc:492] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.083655 5839 ts_tablet_manager.cc:1367] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.091s user 0.001s sys 0.029s
W20240507 08:12:21.083892 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8d1a4e05273b4f8c83dc8c7825a39310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d1a4e05273b4f8c83dc8c7825a39310 already in progress: creating tablet
I20240507 08:12:21.084334 5839 raft_consensus.cc:348] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.084774 5839 raft_consensus.cc:374] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.084836 5839 raft_consensus.cc:729] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.084731 5841 tablet_replica.cc:330] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.085081 5839 consensus_queue.cc:260] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.085597 5839 ts_tablet_manager.cc:1398] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.085850 5839 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:21.085204 5841 raft_consensus.cc:2229] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.086107 5841 raft_consensus.cc:2258] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.087374 5839 tablet_bootstrap.cc:654] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.089296 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 04c410e738234de590b7aecd80aebee8 (DEFAULT_TABLE table=default.Table_3df807895f4648f18a0eb83bb6307de0 [id=cccef4ba6d2a4f45a18738e2f49020d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.089804 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c410e738234de590b7aecd80aebee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.090426 5445 raft_consensus.cc:1261] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Refusing update from remote peer 1cbb887175f345ada2433adaec906fd4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240507 08:12:21.091205 7579 consensus_queue.cc:1035] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.091864 5839 tablet_bootstrap.cc:492] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.091976 5839 ts_tablet_manager.cc:1367] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
W20240507 08:12:21.093189 5656 consensus_peers.cc:485] T 167ee9cbc9174b82824c468e098472a6 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 167ee9cbc9174b82824c468e098472a6. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.093838 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 04c410e738234de590b7aecd80aebee8 (DEFAULT_TABLE table=default.Table_3df807895f4648f18a0eb83bb6307de0 [id=cccef4ba6d2a4f45a18738e2f49020d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.094219 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04c410e738234de590b7aecd80aebee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.097736 5839 raft_consensus.cc:348] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.097909 5839 raft_consensus.cc:374] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.097954 5839 raft_consensus.cc:729] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.098141 5839 consensus_queue.cc:260] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.098608 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad00dca247c4e45b617a47ec6990a63 (DEFAULT_TABLE table=default.Table_5b56cd2bad1d4b3db77c63f023cd204c [id=2e52234ba1a246dbb7ce1b4a7fddf194]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.098639 5839 ts_tablet_manager.cc:1398] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
W20240507 08:12:21.099987 5378 consensus_peers.cc:485] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.101339 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad00dca247c4e45b617a47ec6990a63 (DEFAULT_TABLE table=default.Table_5b56cd2bad1d4b3db77c63f023cd204c [id=2e52234ba1a246dbb7ce1b4a7fddf194]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.101722 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad00dca247c4e45b617a47ec6990a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.104852 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 11bc9baf48c54c388015cf99a80b7829 (DEFAULT_TABLE table=default.Table_74a33830b6b843e6b28ba70ec7b0d05d [id=e506293d80844be8be28d8c97da27998]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.106670 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 3ad00dca247c4e45b617a47ec6990a63 (DEFAULT_TABLE table=default.Table_5b56cd2bad1d4b3db77c63f023cd204c [id=2e52234ba1a246dbb7ce1b4a7fddf194]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.107117 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad00dca247c4e45b617a47ec6990a63. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.109194 5654 consensus_peers.cc:485] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.109830 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 11bc9baf48c54c388015cf99a80b7829 (DEFAULT_TABLE table=default.Table_74a33830b6b843e6b28ba70ec7b0d05d [id=e506293d80844be8be28d8c97da27998]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.110343 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bc9baf48c54c388015cf99a80b7829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.111850 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 11bc9baf48c54c388015cf99a80b7829 (DEFAULT_TABLE table=default.Table_74a33830b6b843e6b28ba70ec7b0d05d [id=e506293d80844be8be28d8c97da27998]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.112273 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bc9baf48c54c388015cf99a80b7829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.111943 7079 ts_tablet_manager.cc:1888] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:21.115264 7079 log.cc:1196] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/89db2847c09b4c74b809bd13f78d0b8b
I20240507 08:12:21.115633 7079 ts_tablet_manager.cc:1909] T 89db2847c09b4c74b809bd13f78d0b8b P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:21.115937 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3ad00dca247c4e45b617a47ec6990a63. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.116492 5839 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:21.118225 5839 tablet_bootstrap.cc:654] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.111485 7762 raft_consensus.cc:482] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.119241 7762 raft_consensus.cc:504] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.120441 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fac82e12d0437e950d01dbc4117efa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.120618 5717 raft_consensus.cc:2454] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:21.121137 5377 leader_election.cc:302] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:21.121480 7722 raft_consensus.cc:2781] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.121553 7722 raft_consensus.cc:482] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.121647 7722 raft_consensus.cc:3037] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.124816 7722 raft_consensus.cc:504] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:21.125512 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 75b716bdf47941049a6e247b4518019e consensus state has no leader
I20240507 08:12:21.126051 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 99a243c9597348bb96f10f8481200a7d
I20240507 08:12:21.126523 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fac82e12d0437e950d01dbc4117efa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.127210 5378 leader_election.cc:341] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.127430 7722 leader_election.cc:288] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.127791 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fac82e12d0437e950d01dbc4117efa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.128463 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a243c9597348bb96f10f8481200a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:21.128527 5378 leader_election.cc:341] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.120837 7762 leader_election.cc:288] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.129245 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4fac82e12d0437e950d01dbc4117efa" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:21.129418 5717 raft_consensus.cc:3037] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.129196 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a243c9597348bb96f10f8481200a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.132781 5717 raft_consensus.cc:2454] T d4fac82e12d0437e950d01dbc4117efa P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:21.133147 5377 leader_election.cc:302] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.133628 7762 raft_consensus.cc:2781] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.133868 7762 raft_consensus.cc:686] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:21.134281 7762 consensus_queue.cc:237] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.136837 5841 ts_tablet_manager.cc:1875] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.137101 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a0de11443df84d3fa973c9e15e8370d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.137677 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0de11443df84d3fa973c9e15e8370d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.138085 5255 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f35aa593fc4c45319217c4ce761d30f9" } modify_external_catalogs: true
I20240507 08:12:21.138239 5255 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f35aa593fc4c45319217c4ce761d30f9" } modify_external_catalogs: true
I20240507 08:12:21.138887 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dac9feb69fbd4fab970744ba42ac0d51" } modify_external_catalogs: true
I20240507 08:12:21.139014 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_dac9feb69fbd4fab970744ba42ac0d51" } modify_external_catalogs: true
I20240507 08:12:21.139983 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_512dbcaed1c74258922227c3af764ede" } modify_external_catalogs: true
I20240507 08:12:21.140112 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_512dbcaed1c74258922227c3af764ede" } modify_external_catalogs: true
I20240507 08:12:21.140950 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_c868ee60064b48408aed1525225921ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.141330 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_bf8e47d12f2f469088138f7e5059fd28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.141464 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c868ee60064b48408aed1525225921ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20240507 08:12:21.140534 7201 log.cc:924] Time spent T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
W20240507 08:12:21.141745 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf8e47d12f2f469088138f7e5059fd28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.142748 5261 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_0601397159fe47908438195478e09248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.143151 5261 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0601397159fe47908438195478e09248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.143687 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 99a243c9597348bb96f10f8481200a7d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.143535 5263 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_3076d02d7ffb4704b6a793f29705bdeb [id=2435a4b9c9a7464aafa19339bf7c9277] alter complete (version 5)
I20240507 08:12:21.159739 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 055a30bfc61b4ff2be9f2418ca59b030 with delete_type TABLET_DATA_DELETED (replaced by tablet 38ce7a166001486197b78f8b02a437ed) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.160142 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 11bc9baf48c54c388015cf99a80b7829. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.160513 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet f2de26190bc44916809856d84ad68345 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:08 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.161175 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b1fadbfd330b4e699e151f83125da281 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.161783 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 84212d6cb2c742c1a0f3fdae8be8ebe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.162456 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 53bc9bf1c04e4015a152cb750184c9ce with delete_type TABLET_DATA_DELETED (replaced by tablet f416d08748ac483ab0ce98240e9eb036) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.163093 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a015bf84a454d49b6497c16ed15c84a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:56 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.163851 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f01e3311a8b4553a31914922fb08d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:55 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.164458 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 767905763ba14cb8bbc52151ae9f17e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.165118 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b8ebbcf0d5a54d63a96a9374e839af83 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.168094 5840 tablet_bootstrap.cc:492] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.168229 5840 ts_tablet_manager.cc:1367] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.091s user 0.003s sys 0.000s
I20240507 08:12:21.168927 5840 raft_consensus.cc:348] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.169112 5840 raft_consensus.cc:374] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.169162 5840 raft_consensus.cc:729] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.169375 5840 consensus_queue.cc:260] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.170068 5840 ts_tablet_manager.cc:1398] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.172230 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_57599205c4f54bf09ca2afcc8efc15ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.172814 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57599205c4f54bf09ca2afcc8efc15ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.173626 5255 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 5ae07896c98b4d50b3cfae208b46ea76
I20240507 08:12:21.174222 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.174340 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_ec9b573b9b3041458e4e9844825c7ec7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.174774 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec9b573b9b3041458e4e9844825c7ec7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.175459 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.176409 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ae07896c98b4d50b3cfae208b46ea76 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.179534 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 787a8804fc8f40c882abd84d6688ea95 (DEFAULT_TABLE table=default.Table_c60abdd82927451d901f94b4868f5c19 [id=91301e389e5d42a9aa261a6f4a42bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.180569 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab5762b4704f689744069308c8cb11 (DEFAULT_TABLE table=default.Table_c795e3e1dcbe4533b864459dda2a2ecc [id=40edbc134d2841c08f5d3d9080f7d082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.180924 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab5762b4704f689744069308c8cb11. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.185431 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 787a8804fc8f40c882abd84d6688ea95 (DEFAULT_TABLE table=default.Table_c60abdd82927451d901f94b4868f5c19 [id=91301e389e5d42a9aa261a6f4a42bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.186638 5840 tablet_bootstrap.cc:492] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.186717 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787a8804fc8f40c882abd84d6688ea95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.189158 5840 tablet_bootstrap.cc:654] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.193564 7732 tablet.cc:1722] T 66247e08dfdf47f69ba24b074c8080f0 P 1cbb887175f345ada2433adaec906fd4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20240507 08:12:21.194386 5656 consensus_peers.cc:485] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3ccc3e67eeb34c42b36edde7bacd23a5. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.195019 7790 tablet.cc:1722] T 66247e08dfdf47f69ba24b074c8080f0 P dc665657d5b34fd7aec884e0f2668e11: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20240507 08:12:21.200753 5839 tablet_bootstrap.cc:492] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.200749 5581 raft_consensus.cc:3037] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.201244 5839 ts_tablet_manager.cc:1367] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.020s
I20240507 08:12:21.201993 5839 raft_consensus.cc:348] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.202183 5839 raft_consensus.cc:374] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.202230 5839 raft_consensus.cc:729] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.202252 5494 heartbeater.cc:347] Connected to a master server at 127.4.246.253:43459
I20240507 08:12:21.202441 5839 consensus_queue.cc:260] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.202948 5839 ts_tablet_manager.cc:1398] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:21.203186 5839 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:21.204505 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 787a8804fc8f40c882abd84d6688ea95 (DEFAULT_TABLE table=default.Table_c60abdd82927451d901f94b4868f5c19 [id=91301e389e5d42a9aa261a6f4a42bba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.204962 5839 tablet_bootstrap.cc:654] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.206308 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787a8804fc8f40c882abd84d6688ea95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.211858 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7ef0ec9add454eb28c8bb9725ee5274f" } new_table_name: "default.Table_2f19dd0d18814f168ad295de7f39d195" modify_external_catalogs: true
I20240507 08:12:21.212898 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 787a8804fc8f40c882abd84d6688ea95. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.214896 7762 raft_consensus.cc:482] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.215008 7762 raft_consensus.cc:504] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.215358 5840 tablet_bootstrap.cc:492] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.215466 5840 ts_tablet_manager.cc:1367] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20240507 08:12:21.215555 7762 leader_election.cc:288] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.215646 5257 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_e026d41a44174419bfcc4a533d8faf3c [id=5bef63e11c734867ab4c67ca05d8e2e5] alter complete (version 1)
I20240507 08:12:21.216243 5840 raft_consensus.cc:348] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.216418 5840 raft_consensus.cc:374] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.216471 5840 raft_consensus.cc:729] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.216686 5840 consensus_queue.cc:260] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.217226 5840 ts_tablet_manager.cc:1398] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.217445 5840 tablet_bootstrap.cc:492] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.217440 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_5b0f0f6f249a4ceb83421dc56c43921e" } new_table_name: "default.Table_0d4af04e5f9f4906a49e5a248162b4c3" modify_external_catalogs: true
I20240507 08:12:21.218274 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506fb6f8445a402eb9fbc81c8019cc8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:21.218851 5378 leader_election.cc:341] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.218942 5840 tablet_bootstrap.cc:654] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.219059 5262 catalog_manager.cc:5766] Not sending DeleteTablet requests; no consensus state for tablet 01152e5dd4524124bef1932a535a969b
I20240507 08:12:21.219386 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_4cfd91df9cb946a391ff086b5ddee69a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.219857 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4cfd91df9cb946a391ff086b5ddee69a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.221318 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac
I20240507 08:12:21.221931 5256 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_858753a9d5084a709cac3cf93887cf25" } new_table_name: "default.Table_ab2bf5b76a214c228d43045ec8d1ed08" modify_external_catalogs: true
I20240507 08:12:21.222096 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:21.222723 5656 consensus_peers.cc:485] T b60ec15a83934999871425ef9a64ec72 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.223683 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_af114d78455749ce9330a056f6505e04" } modify_external_catalogs: true
I20240507 08:12:21.223821 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_af114d78455749ce9330a056f6505e04" } modify_external_catalogs: true
I20240507 08:12:21.223989 5839 tablet_bootstrap.cc:492] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.224117 5839 ts_tablet_manager.cc:1367] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20240507 08:12:21.224785 5839 raft_consensus.cc:348] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.224974 5839 raft_consensus.cc:374] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.225032 5839 raft_consensus.cc:729] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.225205 5839 consensus_queue.cc:260] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.225720 5839 ts_tablet_manager.cc:1398] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:21.225976 5839 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:21.226650 5417 tablet_service.cc:1514] Processing DeleteTablet for tablet 093fb3986a674b93a435408c7ec85500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.226694 5416 tablet_service.cc:1514] Processing DeleteTablet for tablet 8005100f64fd49a3ac29d34aa728fdfb with delete_type TABLET_DATA_DELETED (replaced by tablet 43fa551f9e8b4f0bb3a53bf2917a08c4) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.226761 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.227334 5839 tablet_bootstrap.cc:654] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.231753 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet b31fdadee60e4f5e94e92c536e18774d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.231789 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 093fb3986a674b93a435408c7ec85500 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:14 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.231879 5689 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.231937 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f43a80d38ed4788a13421b245fe6e8b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.232082 5687 tablet_service.cc:1514] Processing DeleteTablet for tablet d4fac82e12d0437e950d01dbc4117efa with delete_type TABLET_DATA_DELETED (replaced by tablet 11e779c5c97c44e5bcc7c0eb0194fd95) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.232462 5584 raft_consensus.cc:3037] T 3b9429e5cc274198974b27c40c7e4e37 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.234542 5264 catalog_manager.cc:5484] T ec08c0d8a1ab444489340415303a5739 P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:21.234728 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506fb6f8445a402eb9fbc81c8019cc8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.234897 5717 raft_consensus.cc:2454] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:21.235111 5264 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet ec08c0d8a1ab444489340415303a5739 (table default.Table_73aa505d7d4a4389bbd269ee956f5d8a [id=c4b899bf5b0244ac897da574bda1457d]). Expected version 1 got 0
I20240507 08:12:21.235229 5377 leader_election.cc:302] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.235491 7762 raft_consensus.cc:2781] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.235586 7762 raft_consensus.cc:482] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.235657 7762 raft_consensus.cc:3037] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.237761 5263 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
I20240507 08:12:21.238052 7762 raft_consensus.cc:504] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.238793 7762 leader_election.cc:288] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.239344 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506fb6f8445a402eb9fbc81c8019cc8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:21.239688 5559 tablet_service.cc:1514] Processing DeleteTablet for tablet 99e4514039304f0696c69a057db0ea6e with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:59 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:21.239913 5378 leader_election.cc:341] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.240676 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506fb6f8445a402eb9fbc81c8019cc8b" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:21.241888 5552 tablet_service.cc:1514] Processing DeleteTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.240938 5717 raft_consensus.cc:3037] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.246238 5717 raft_consensus.cc:2454] T 506fb6f8445a402eb9fbc81c8019cc8b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:21.246628 5377 leader_election.cc:302] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.247004 7762 raft_consensus.cc:2781] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.247208 7762 raft_consensus.cc:686] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:21.247419 7762 consensus_queue.cc:237] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.247853 5839 tablet_bootstrap.cc:492] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.248023 5839 ts_tablet_manager.cc:1367] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20240507 08:12:21.248843 5839 raft_consensus.cc:348] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.249075 5839 raft_consensus.cc:374] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.249187 5839 raft_consensus.cc:729] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.249508 5843 tablet_bootstrap.cc:492] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.249711 5843 ts_tablet_manager.cc:1367] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.213s user 0.012s sys 0.053s
I20240507 08:12:21.250401 5843 raft_consensus.cc:348] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.250815 5843 raft_consensus.cc:374] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.250916 5843 raft_consensus.cc:729] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.251169 5843 consensus_queue.cc:260] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.251353 5839 consensus_queue.cc:260] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.251796 5843 ts_tablet_manager.cc:1398] T f64e338784704312b986762a143dda4c P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.251829 5839 ts_tablet_manager.cc:1398] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20240507 08:12:21.252115 5843 tablet_bootstrap.cc:492] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.257592 5839 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:21.258101 5843 tablet_bootstrap.cc:654] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.261147 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1306) took 2177 ms. Trace:
I20240507 08:12:21.261243 7079 rpcz_store.cc:276] 0507 08:12:19.084127 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.084207 (+ 80us) service_pool.cc:225] Handling call
0507 08:12:20.797490 (+1713283us) ts_tablet_manager.cc:1120] Deleting tablet 89db2847c09b4c74b809bd13f78d0b8b
0507 08:12:20.797498 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:20.976027 (+178529us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.110478 (+134451us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.261132 (+150654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1713095}
I20240507 08:12:21.261690 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]) successfully deleted
I20240507 08:12:21.262843 5843 tablet_bootstrap.cc:492] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
W20240507 08:12:21.262871 5654 consensus_peers.cc:485] T 156bf04b57724134a45fa9fc477596c5 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.263414 5839 tablet_bootstrap.cc:654] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.263777 5843 ts_tablet_manager.cc:1367] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
W20240507 08:12:21.265055 5656 consensus_peers.cc:485] T 3fce98f472ef46589dbea6b6aefa6572 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3fce98f472ef46589dbea6b6aefa6572. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.265878 5843 raft_consensus.cc:348] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.269824 5843 raft_consensus.cc:374] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.270529 5840 tablet_bootstrap.cc:492] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.270701 5840 ts_tablet_manager.cc:1367] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.008s
I20240507 08:12:21.271441 5840 raft_consensus.cc:348] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.271678 5840 raft_consensus.cc:374] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.271781 5840 raft_consensus.cc:729] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.272007 5840 consensus_queue.cc:260] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.272703 5840 ts_tablet_manager.cc:1398] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.272998 5840 tablet_bootstrap.cc:492] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.274617 5840 tablet_bootstrap.cc:654] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.269943 5843 raft_consensus.cc:729] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.274955 5843 consensus_queue.cc:260] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.275651 5843 ts_tablet_manager.cc:1398] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20240507 08:12:21.268440 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab5762b4704f689744069308c8cb11 (DEFAULT_TABLE table=default.Table_c795e3e1dcbe4533b864459dda2a2ecc [id=40edbc134d2841c08f5d3d9080f7d082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.276468 5843 tablet_bootstrap.cc:492] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.266518 5841 ts_tablet_manager.cc:1888] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.276564 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab5762b4704f689744069308c8cb11. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.276549 5654 consensus_peers.cc:485] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.276612 5841 log.cc:1196] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b60ec15a83934999871425ef9a64ec72
I20240507 08:12:21.276937 5841 ts_tablet_manager.cc:1909] T b60ec15a83934999871425ef9a64ec72 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.278780 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1305) took 3219 ms. Trace:
I20240507 08:12:21.278853 5841 rpcz_store.cc:276] 0507 08:12:18.059719 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.059777 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:21.084689 (+3024912us) ts_tablet_manager.cc:1120] Deleting tablet b60ec15a83934999871425ef9a64ec72
0507 08:12:21.084698 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.212265 (+127567us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.266444 (+ 54179us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.278763 (+ 12319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017858}
I20240507 08:12:21.279260 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1314) took 3147 ms. Trace:
I20240507 08:12:21.279331 5841 rpcz_store.cc:276] 0507 08:12:18.132025 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.134622 (+ 2597us) service_pool.cc:225] Handling call
0507 08:12:21.279150 (+3144528us) ts_tablet_manager.cc:1120] Deleting tablet f859146c376c4b1eb52bff953f05837b
0507 08:12:21.279159 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.279245 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3144318}
I20240507 08:12:21.280277 5843 tablet_bootstrap.cc:654] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.281517 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b60ec15a83934999871425ef9a64ec72 (table default.Table_aa2b432cc7ef4604b737ade9696fe60f [id=46a6824713664be39257556ade2fe5e3]) successfully deleted
W20240507 08:12:21.281810 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet f859146c376c4b1eb52bff953f05837b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f859146c376c4b1eb52bff953f05837b already in progress: creating tablet
I20240507 08:12:21.282718 7579 consensus_queue.cc:1035] T 449d6c8270184ce2918cee8ce731f788 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.280537 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1315) took 3112 ms. Trace:
I20240507 08:12:21.285629 5841 rpcz_store.cc:276] 0507 08:12:18.167895 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.167965 (+ 70us) service_pool.cc:225] Handling call
0507 08:12:21.280443 (+3112478us) ts_tablet_manager.cc:1120] Deleting tablet 636e8f123b9540f3afedf02e4df5dfb5
0507 08:12:21.280452 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.280523 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3112299}
I20240507 08:12:21.285936 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1349) took 2557 ms. Trace:
I20240507 08:12:21.286006 5841 rpcz_store.cc:276] 0507 08:12:18.728612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.789923 (+ 61311us) service_pool.cc:225] Handling call
0507 08:12:21.285849 (+2495926us) ts_tablet_manager.cc:1120] Deleting tablet 04a14cf04d7a4ce3a2df939d0db515b4
0507 08:12:21.285856 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.285923 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428182}
W20240507 08:12:21.286103 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 636e8f123b9540f3afedf02e4df5dfb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 636e8f123b9540f3afedf02e4df5dfb5 already in progress: creating tablet
I20240507 08:12:21.286295 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1350) took 2545 ms. Trace:
I20240507 08:12:21.286360 5841 rpcz_store.cc:276] 0507 08:12:18.741103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.857696 (+116593us) service_pool.cc:225] Handling call
0507 08:12:21.286216 (+2428520us) ts_tablet_manager.cc:1120] Deleting tablet 5aca2077b76b49e3bf9daae57dc09b8f
0507 08:12:21.286223 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.286283 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428373}
I20240507 08:12:21.286604 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1351) took 2542 ms. Trace:
I20240507 08:12:21.286671 5841 rpcz_store.cc:276] 0507 08:12:18.744207 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.857862 (+113655us) service_pool.cc:225] Handling call
0507 08:12:21.286526 (+2428664us) ts_tablet_manager.cc:1120] Deleting tablet ffc89e00efaa4bde9242be40cf409c59
0507 08:12:21.286532 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.286592 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428545}
W20240507 08:12:21.286753 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 04a14cf04d7a4ce3a2df939d0db515b4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 04a14cf04d7a4ce3a2df939d0db515b4 already in progress: creating tablet
I20240507 08:12:21.286927 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1352) took 2540 ms. Trace:
W20240507 08:12:21.287041 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5aca2077b76b49e3bf9daae57dc09b8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5aca2077b76b49e3bf9daae57dc09b8f already in progress: creating tablet
I20240507 08:12:21.287003 5841 rpcz_store.cc:276] 0507 08:12:18.746503 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.857994 (+111491us) service_pool.cc:225] Handling call
0507 08:12:21.286848 (+2428854us) ts_tablet_manager.cc:1120] Deleting tablet 120bce4b20714fc1a0e1237abc00d5bb
0507 08:12:21.286857 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.286916 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428736}
I20240507 08:12:21.287281 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1353) took 2426 ms. Trace:
I20240507 08:12:21.287362 5841 rpcz_store.cc:276] 0507 08:12:18.861210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.873618 (+ 12408us) service_pool.cc:225] Handling call
0507 08:12:21.287203 (+2413585us) ts_tablet_manager.cc:1120] Deleting tablet fdffec6ce0da4c409466783bd76a6b4b
0507 08:12:21.287211 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.287268 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413343}
I20240507 08:12:21.287566 5841 tablet_replica.cc:330] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.287725 5841 raft_consensus.cc:2229] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.287850 5841 raft_consensus.cc:2258] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240507 08:12:21.287835 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ffc89e00efaa4bde9242be40cf409c59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ffc89e00efaa4bde9242be40cf409c59 already in progress: creating tablet
W20240507 08:12:21.288053 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 120bce4b20714fc1a0e1237abc00d5bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 120bce4b20714fc1a0e1237abc00d5bb already in progress: creating tablet
W20240507 08:12:21.288278 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet fdffec6ce0da4c409466783bd76a6b4b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fdffec6ce0da4c409466783bd76a6b4b already in progress: creating tablet
I20240507 08:12:21.288311 5841 ts_tablet_manager.cc:1875] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.290362 5840 tablet_bootstrap.cc:492] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.290915 5840 ts_tablet_manager.cc:1367] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.001s
I20240507 08:12:21.291731 5840 raft_consensus.cc:348] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.291898 5840 raft_consensus.cc:374] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.291950 5840 raft_consensus.cc:729] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.292150 5840 consensus_queue.cc:260] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.292716 5840 ts_tablet_manager.cc:1398] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:21.292958 5840 tablet_bootstrap.cc:492] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.294426 5840 tablet_bootstrap.cc:654] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.294862 7079 tablet_replica.cc:330] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:21.301720 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_891ce74b91c148c18a2e8e9ba4f45e95"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.301985 5843 tablet_bootstrap.cc:492] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.302102 5843 ts_tablet_manager.cc:1367] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
W20240507 08:12:21.302394 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_891ce74b91c148c18a2e8e9ba4f45e95 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.302801 5843 raft_consensus.cc:348] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.303524 5843 raft_consensus.cc:374] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.303583 5843 raft_consensus.cc:729] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.303791 5843 consensus_queue.cc:260] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.304351 7579 raft_consensus.cc:482] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.304476 7579 raft_consensus.cc:504] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.305122 7579 leader_election.cc:288] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.305713 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.305887 5445 raft_consensus.cc:2454] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:21.305930 5843 ts_tablet_manager.cc:1398] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240507 08:12:21.306187 5843 tablet_bootstrap.cc:492] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.306257 5656 leader_election.cc:302] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:21.306483 7579 raft_consensus.cc:2781] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.306547 7579 raft_consensus.cc:482] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.306603 7579 raft_consensus.cc:3037] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.307536 5843 tablet_bootstrap.cc:654] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.298806 7720 raft_consensus.cc:482] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.308660 7720 raft_consensus.cc:504] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.309316 7720 leader_election.cc:288] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.305409 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.311017 7720 raft_consensus.cc:482] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.311172 7720 raft_consensus.cc:504] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.311693 7720 leader_election.cc:288] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.312094 5841 ts_tablet_manager.cc:1888] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.312186 5841 log.cc:1196] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/6def4bf6a7554a4090a39b1823c4097c
I20240507 08:12:21.312436 5841 ts_tablet_manager.cc:1909] T 6def4bf6a7554a4090a39b1823c4097c P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:21.312785 5517 leader_election.cc:341] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 4e5439844f384513b4719cfe4f4e5d9b
I20240507 08:12:21.313206 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.313549 7079 raft_consensus.cc:2229] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.314092 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 41ab5762b4704f689744069308c8cb11 (DEFAULT_TABLE table=default.Table_c795e3e1dcbe4533b864459dda2a2ecc [id=40edbc134d2841c08f5d3d9080f7d082]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.314193 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c4cd7b56d4e468eb754d9653117d023" candidate_uuid: "fe673739d8084e7da5d7edd722e857d8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.314431 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41ab5762b4704f689744069308c8cb11. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.314672 5654 leader_election.cc:341] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.315902 5517 leader_election.cc:302] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.316326 7720 raft_consensus.cc:2726] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.309126 7579 raft_consensus.cc:504] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.316996 7579 leader_election.cc:288] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.317442 5445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:21.317620 5445 raft_consensus.cc:3037] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.317889 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1348) took 2605 ms. Trace:
I20240507 08:12:21.318010 7762 consensus_queue.cc:1035] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.317965 5841 rpcz_store.cc:276] 0507 08:12:18.711971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:18.788874 (+ 76903us) service_pool.cc:225] Handling call
0507 08:12:21.287529 (+2498655us) ts_tablet_manager.cc:1120] Deleting tablet 6def4bf6a7554a4090a39b1823c4097c
0507 08:12:21.287535 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.294961 (+ 7426us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.311985 (+ 17024us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.317874 (+ 5889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2413440}
I20240507 08:12:21.318562 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 6def4bf6a7554a4090a39b1823c4097c (table default.Table_d228b6629c7349a5843e85ab6a55e3bd [id=a57d9cd5e2d5498f9ae045b87c5ccb79]) successfully deleted
I20240507 08:12:21.318838 5841 tablet_replica.cc:330] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.319003 5841 raft_consensus.cc:2229] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.319159 5841 raft_consensus.cc:2258] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.319618 5841 ts_tablet_manager.cc:1875] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.320012 5445 raft_consensus.cc:2454] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:21.320320 7079 raft_consensus.cc:2258] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.320400 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7eac88c20a49079e6544d7c1d3530c (DEFAULT_TABLE table=default.Table_ae198ba314f04af0ba8dde8c1c3f7839 [id=718a1f25ee2a448b937fd8301eb10a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.320751 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7eac88c20a49079e6544d7c1d3530c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.320710 5520 leader_election.cc:341] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 4e5439844f384513b4719cfe4f4e5d9b
I20240507 08:12:21.320817 5520 leader_election.cc:302] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.321146 7720 raft_consensus.cc:2726] T 4e5439844f384513b4719cfe4f4e5d9b P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.321815 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c1f3dc0a02c948e7ad05ddde8f2d58c1" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:21.322074 5843 tablet_bootstrap.cc:492] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.322180 5843 ts_tablet_manager.cc:1367] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20240507 08:12:21.322459 5656 leader_election.cc:302] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:21.322818 7649 raft_consensus.cc:2781] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.322842 6513 ts_tablet_manager.cc:1888] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.322942 6513 log.cc:1196] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/b8c6f552832a45c4a2e28f40797db8fa
I20240507 08:12:21.322927 7649 raft_consensus.cc:686] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:21.322903 5843 raft_consensus.cc:348] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.323073 5843 raft_consensus.cc:374] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.323117 5843 raft_consensus.cc:729] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.323305 6513 ts_tablet_manager.cc:1909] T b8c6f552832a45c4a2e28f40797db8fa P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.323240 7649 consensus_queue.cc:237] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.323300 5843 consensus_queue.cc:260] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.323556 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1238) took 5043 ms. Trace:
I20240507 08:12:21.323629 6513 rpcz_store.cc:276] 0507 08:12:16.279982 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.322993 (+ 43011us) service_pool.cc:225] Handling call
0507 08:12:20.939577 (+4616584us) ts_tablet_manager.cc:1120] Deleting tablet b8c6f552832a45c4a2e28f40797db8fa
0507 08:12:20.939608 (+ 31us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.090594 (+150986us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.322757 (+232163us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.323541 (+ 784us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4616364}
I20240507 08:12:21.323912 6513 tablet_replica.cc:330] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.323906 5843 ts_tablet_manager.cc:1398] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.324129 5843 tablet_bootstrap.cc:492] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.324105 6513 raft_consensus.cc:2229] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.324411 6513 raft_consensus.cc:2258] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.324659 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet b8c6f552832a45c4a2e28f40797db8fa (table default.Table_6500147915384d518c459e558a9ce1d6 [id=98ca5c705ae24203aa9e3946bcbc3bbf]) successfully deleted
W20240507 08:12:21.324694 5654 leader_election.cc:341] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.325506 5843 tablet_bootstrap.cc:654] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.326668 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet e14487e625374060b118c9cce4500b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:57 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.325928 5410 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7eac88c20a49079e6544d7c1d3530c (DEFAULT_TABLE table=default.Table_ae198ba314f04af0ba8dde8c1c3f7839 [id=718a1f25ee2a448b937fd8301eb10a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.327351 5840 tablet_bootstrap.cc:492] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.327518 5840 ts_tablet_manager.cc:1367] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.002s
I20240507 08:12:21.328267 5840 raft_consensus.cc:348] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.328445 5840 raft_consensus.cc:374] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.328491 5840 raft_consensus.cc:729] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.328698 5840 consensus_queue.cc:260] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.329165 5840 ts_tablet_manager.cc:1398] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:21.329442 5840 tablet_bootstrap.cc:492] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.327778 7720 raft_consensus.cc:482] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.330835 7720 raft_consensus.cc:504] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.331367 7720 leader_election.cc:288] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.332096 5840 tablet_bootstrap.cc:654] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
W20240507 08:12:21.332131 5520 leader_election.cc:341] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: e14487e625374060b118c9cce4500b3f
W20240507 08:12:21.332269 5517 leader_election.cc:341] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: e14487e625374060b118c9cce4500b3f
I20240507 08:12:21.332556 5517 leader_election.cc:302] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.332934 7720 raft_consensus.cc:2726] T e14487e625374060b118c9cce4500b3f P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.336088 7720 raft_consensus.cc:482] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.336278 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7eac88c20a49079e6544d7c1d3530c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.336210 7720 raft_consensus.cc:504] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.336735 7720 leader_election.cc:288] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.337428 5520 leader_election.cc:341] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b8c6f552832a45c4a2e28f40797db8fa
W20240507 08:12:21.337555 5517 leader_election.cc:341] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: b8c6f552832a45c4a2e28f40797db8fa
I20240507 08:12:21.337707 5517 leader_election.cc:302] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.338212 7720 raft_consensus.cc:2726] T b8c6f552832a45c4a2e28f40797db8fa P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.339269 5841 ts_tablet_manager.cc:1888] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:21.339381 5841 log.cc:1196] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/89db2847c09b4c74b809bd13f78d0b8b
I20240507 08:12:21.339723 5841 ts_tablet_manager.cc:1909] T 89db2847c09b4c74b809bd13f78d0b8b P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.340102 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1354) took 2256 ms. Trace:
I20240507 08:12:21.340201 5841 rpcz_store.cc:276] 0507 08:12:19.083667 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.083718 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:21.318797 (+2235079us) ts_tablet_manager.cc:1120] Deleting tablet 89db2847c09b4c74b809bd13f78d0b8b
0507 08:12:21.318805 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.328085 (+ 9280us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.339184 (+ 11099us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.340079 (+ 895us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2234870}
I20240507 08:12:21.340799 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1356) took 2033 ms. Trace:
I20240507 08:12:21.340684 5260 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7a766eb1103b48a79a8e123eab7f8920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.340880 5841 rpcz_store.cc:276] 0507 08:12:19.307805 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.307898 (+ 93us) service_pool.cc:225] Handling call
0507 08:12:21.340668 (+2032770us) ts_tablet_manager.cc:1120] Deleting tablet a905e345412541558b30c629a2eb86c5
0507 08:12:21.340681 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.340782 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2032583}
W20240507 08:12:21.341284 5260 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a766eb1103b48a79a8e123eab7f8920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.341346 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1355) took 2034 ms. Trace:
I20240507 08:12:21.341413 5841 rpcz_store.cc:276] 0507 08:12:19.307282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.307536 (+ 254us) service_pool.cc:225] Handling call
0507 08:12:21.341254 (+2033718us) ts_tablet_manager.cc:1120] Deleting tablet 01820ecad55a4097bc5917a8f7949fe7
0507 08:12:21.341262 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.341330 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2033127}
I20240507 08:12:21.341853 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1360) took 2022 ms. Trace:
I20240507 08:12:21.341917 5841 rpcz_store.cc:276] 0507 08:12:19.319152 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.321062 (+ 1910us) service_pool.cc:225] Handling call
0507 08:12:21.341757 (+2020695us) ts_tablet_manager.cc:1120] Deleting tablet c1935dd9e2a34c49bb3e0021f46bc765
0507 08:12:21.341764 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.341837 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020516}
I20240507 08:12:21.342453 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 89db2847c09b4c74b809bd13f78d0b8b (table default.Table_f218240b99734c72a1ef75b6e7bb6534 [id=f21341e3ea0a49ebacc9309604184464]) successfully deleted
W20240507 08:12:21.342672 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet a905e345412541558b30c629a2eb86c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a905e345412541558b30c629a2eb86c5 already in progress: creating tablet
I20240507 08:12:21.342707 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1366) took 2023 ms. Trace:
I20240507 08:12:21.342773 5841 rpcz_store.cc:276] 0507 08:12:19.319560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.322467 (+ 2907us) service_pool.cc:225] Handling call
0507 08:12:21.342614 (+2020147us) ts_tablet_manager.cc:1120] Deleting tablet d3f120d9fbf542d39508903883ecb5e6
0507 08:12:21.342622 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.342692 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020006}
W20240507 08:12:21.342888 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 01820ecad55a4097bc5917a8f7949fe7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01820ecad55a4097bc5917a8f7949fe7 already in progress: creating tablet
W20240507 08:12:21.343039 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c1935dd9e2a34c49bb3e0021f46bc765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1935dd9e2a34c49bb3e0021f46bc765 already in progress: creating tablet
W20240507 08:12:21.343227 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d3f120d9fbf542d39508903883ecb5e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3f120d9fbf542d39508903883ecb5e6 already in progress: creating tablet
I20240507 08:12:21.343536 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1359) took 2024 ms. Trace:
I20240507 08:12:21.343617 5841 rpcz_store.cc:276] 0507 08:12:19.319085 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.328398 (+ 9313us) service_pool.cc:225] Handling call
0507 08:12:21.343444 (+2015046us) ts_tablet_manager.cc:1120] Deleting tablet aae2f13901a146b7a2019709b28930e9
0507 08:12:21.343452 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.343520 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2014898}
W20240507 08:12:21.343953 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet aae2f13901a146b7a2019709b28930e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aae2f13901a146b7a2019709b28930e9 already in progress: creating tablet
I20240507 08:12:21.344290 7579 consensus_queue.cc:1035] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.344694 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1370) took 1991 ms. Trace:
I20240507 08:12:21.344774 5841 rpcz_store.cc:276] 0507 08:12:19.352902 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.352961 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:21.344620 (+1991659us) ts_tablet_manager.cc:1120] Deleting tablet b8c6f552832a45c4a2e28f40797db8fa
0507 08:12:21.344629 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.344680 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1991472}
W20240507 08:12:21.345149 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b8c6f552832a45c4a2e28f40797db8fa because the tablet was not found. No further retry: Not found: Tablet not found: b8c6f552832a45c4a2e28f40797db8fa
I20240507 08:12:21.348333 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1369) took 2001 ms. Trace:
I20240507 08:12:21.348351 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e5439844f384513b4719cfe4f4e5d9b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.348635 5841 rpcz_store.cc:276] 0507 08:12:19.346417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.357940 (+ 11523us) service_pool.cc:225] Handling call
0507 08:12:21.348261 (+1990321us) ts_tablet_manager.cc:1120] Deleting tablet 99e4514039304f0696c69a057db0ea6e
0507 08:12:21.348267 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.348318 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1989213}
I20240507 08:12:21.349161 7079 ts_tablet_manager.cc:1875] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.350078 5251 catalog_manager.cc:4797] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 99e4514039304f0696c69a057db0ea6e because the tablet was not found. No further retry: Not found: Tablet not found: 99e4514039304f0696c69a057db0ea6e
I20240507 08:12:21.349689 5843 tablet_bootstrap.cc:492] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.350505 5843 ts_tablet_manager.cc:1367] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20240507 08:12:21.351524 5843 raft_consensus.cc:348] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.352056 5843 raft_consensus.cc:374] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.352182 5843 raft_consensus.cc:729] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.352638 5843 consensus_queue.cc:260] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.353559 5843 ts_tablet_manager.cc:1398] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240507 08:12:21.353973 5843 tablet_bootstrap.cc:492] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.355909 5843 tablet_bootstrap.cc:654] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.358060 5687 tablet_service.cc:1467] Processing CreateTablet for tablet 1f7eac88c20a49079e6544d7c1d3530c (DEFAULT_TABLE table=default.Table_ae198ba314f04af0ba8dde8c1c3f7839 [id=718a1f25ee2a448b937fd8301eb10a42]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.358428 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f7eac88c20a49079e6544d7c1d3530c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.354260 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1371) took 1930 ms. Trace:
I20240507 08:12:21.359652 5841 rpcz_store.cc:276] 0507 08:12:19.423954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.484431 (+ 60477us) service_pool.cc:225] Handling call
0507 08:12:21.354130 (+1869699us) ts_tablet_manager.cc:1120] Deleting tablet ece0e5eae0d949149dd3bd3f4bbd4ca7
0507 08:12:21.354138 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.354241 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1868550}
W20240507 08:12:21.360086 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 already in progress: creating tablet
I20240507 08:12:21.360390 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1372) took 1861 ms. Trace:
I20240507 08:12:21.360467 5841 rpcz_store.cc:276] 0507 08:12:19.499383 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.499445 (+ 62us) service_pool.cc:225] Handling call
0507 08:12:21.360297 (+1860852us) ts_tablet_manager.cc:1120] Deleting tablet e797fee2c43c4461a5cb9e2f0a2bb0fc
0507 08:12:21.360306 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.360376 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1860651}
W20240507 08:12:21.360819 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e797fee2c43c4461a5cb9e2f0a2bb0fc already in progress: creating tablet
I20240507 08:12:21.361413 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1373) took 1860 ms. Trace:
I20240507 08:12:21.361478 5841 rpcz_store.cc:276] 0507 08:12:19.501178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.501236 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:21.361322 (+1860086us) ts_tablet_manager.cc:1120] Deleting tablet b82da7a22025499b928406579ae43a50
0507 08:12:21.361330 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.361397 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859811}
I20240507 08:12:21.361646 7750 tablet.cc:1722] T 6f3bdb777b974524b5673a3f02883107 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20240507 08:12:21.361838 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b82da7a22025499b928406579ae43a50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b82da7a22025499b928406579ae43a50 already in progress: creating tablet
I20240507 08:12:21.361907 5553 tablet_service.cc:1467] Processing CreateTablet for tablet b6a9017c700743ba933c62cf1bd178fc (DEFAULT_TABLE table=default.Table_2c24bc292e0e44fba3e26850b6615d06 [id=8034f800a2ed45858cc353fbef9921de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.362098 5841 tablet_replica.cc:330] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.362258 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a9017c700743ba933c62cf1bd178fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.362277 5841 raft_consensus.cc:2229] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.362399 5841 raft_consensus.cc:2258] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.362841 5841 ts_tablet_manager.cc:1875] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.368268 5841 ts_tablet_manager.cc:1888] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.368366 5841 log.cc:1196] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/5f0b399a2d074f4ca3626ff17bd5518e
I20240507 08:12:21.368634 5841 ts_tablet_manager.cc:1909] T 5f0b399a2d074f4ca3626ff17bd5518e P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.368917 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1374) took 1583 ms. Trace:
I20240507 08:12:21.368993 5841 rpcz_store.cc:276] 0507 08:12:19.785656 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.785714 (+ 58us) service_pool.cc:225] Handling call
0507 08:12:21.362059 (+1576345us) ts_tablet_manager.cc:1120] Deleting tablet 5f0b399a2d074f4ca3626ff17bd5518e
0507 08:12:21.362067 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.365680 (+ 3613us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.368204 (+ 2524us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.368901 (+ 697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1484371}
I20240507 08:12:21.369339 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 5f0b399a2d074f4ca3626ff17bd5518e (table default.Table_f7d324ad4a784569bdf917a46203715f [id=0a13053ee3a44ee1a9588eb3ba56b74f]) successfully deleted
I20240507 08:12:21.369529 5584 raft_consensus.cc:3037] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.369776 5840 tablet_bootstrap.cc:492] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.369908 5840 ts_tablet_manager.cc:1367] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.005s
I20240507 08:12:21.371138 5840 raft_consensus.cc:348] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.372814 7720 raft_consensus.cc:482] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.372972 7720 raft_consensus.cc:504] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.373683 7720 leader_election.cc:288] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:21.371328 5840 raft_consensus.cc:374] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.375651 5840 raft_consensus.cc:729] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.375849 5840 consensus_queue.cc:260] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.376461 5840 ts_tablet_manager.cc:1398] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240507 08:12:21.377089 5840 tablet_bootstrap.cc:492] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.378494 5840 tablet_bootstrap.cc:654] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.379978 5841 tablet_replica.cc:330] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.380183 5841 raft_consensus.cc:2229] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.380335 5841 raft_consensus.cc:2258] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.381686 5841 ts_tablet_manager.cc:1875] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.381929 5520 leader_election.cc:341] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 8a015bf84a454d49b6497c16ed15c84a
I20240507 08:12:21.387450 5840 tablet_bootstrap.cc:492] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.387598 5840 ts_tablet_manager.cc:1367] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:21.388260 5840 raft_consensus.cc:348] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.388469 5840 raft_consensus.cc:374] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.388870 5840 raft_consensus.cc:729] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
W20240507 08:12:21.389258 5656 consensus_peers.cc:485] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89db2847c09b4c74b809bd13f78d0b8b. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.389163 5840 consensus_queue.cc:260] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.389875 5840 ts_tablet_manager.cc:1398] T 8d6b0ccf73cc46a3a7815e5d09ed7614 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:21.390210 5840 tablet_bootstrap.cc:492] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.391691 5840 tablet_bootstrap.cc:654] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.392625 7750 tablet.cc:1722] T 645ae26bb26f43d298afe98817d88fa4 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:21.393488 5843 tablet_bootstrap.cc:492] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.393682 5843 ts_tablet_manager.cc:1367] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.008s
I20240507 08:12:21.394912 5843 raft_consensus.cc:348] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.395165 5843 raft_consensus.cc:374] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.395252 5843 raft_consensus.cc:729] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.395514 5843 consensus_queue.cc:260] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.396214 5843 ts_tablet_manager.cc:1398] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.396581 5843 tablet_bootstrap.cc:492] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
W20240507 08:12:21.397758 5517 leader_election.cc:341] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 8a015bf84a454d49b6497c16ed15c84a
I20240507 08:12:21.397969 5517 leader_election.cc:302] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.398443 7720 raft_consensus.cc:2726] T 8a015bf84a454d49b6497c16ed15c84a P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.399905 5841 ts_tablet_manager.cc:1888] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.400007 5841 log.cc:1196] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/eec3704b135740f1a906d60f3ea87ffe
I20240507 08:12:21.400288 5841 ts_tablet_manager.cc:1909] T eec3704b135740f1a906d60f3ea87ffe P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.400593 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1375) took 1614 ms. Trace:
I20240507 08:12:21.400671 5841 rpcz_store.cc:276] 0507 08:12:19.785758 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.786010 (+ 252us) service_pool.cc:225] Handling call
0507 08:12:21.379919 (+1593909us) ts_tablet_manager.cc:1120] Deleting tablet eec3704b135740f1a906d60f3ea87ffe
0507 08:12:21.379929 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.384543 (+ 4614us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.399818 (+ 15275us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.400575 (+ 757us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1502166}
I20240507 08:12:21.400889 5840 tablet_bootstrap.cc:492] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.401021 5840 ts_tablet_manager.cc:1367] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20240507 08:12:21.401090 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet eec3704b135740f1a906d60f3ea87ffe (table default.Table_221bfcf09b8e4d5e95e2ab49ad514c10 [id=1345eee5280943ad899d0c66889b16ab]) successfully deleted
W20240507 08:12:21.401700 5654 consensus_peers.cc:485] T d17b6a2f4aa14f2a927fa9ad56a62a5a P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.401762 5840 raft_consensus.cc:348] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.401959 5840 raft_consensus.cc:374] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.402038 5840 raft_consensus.cc:729] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.402115 5843 tablet_bootstrap.cc:654] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.402247 5840 consensus_queue.cc:260] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.402782 5840 ts_tablet_manager.cc:1398] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240507 08:12:21.403053 5840 tablet_bootstrap.cc:492] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.404230 7743 raft_consensus.cc:482] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.404405 5840 tablet_bootstrap.cc:654] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.404366 7743 raft_consensus.cc:504] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.405014 7743 leader_election.cc:288] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.405735 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3397e33b4cc7805a25e7535a383a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.405965 5446 raft_consensus.cc:2454] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:21.406108 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3397e33b4cc7805a25e7535a383a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.406761 5656 leader_election.cc:302] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters:
I20240507 08:12:21.407052 7748 raft_consensus.cc:2781] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.407137 7748 raft_consensus.cc:482] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.407220 7748 raft_consensus.cc:3037] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.409744 5841 tablet_replica.cc:330] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.409942 5841 raft_consensus.cc:2229] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:21.409946 5654 leader_election.cc:341] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.410166 5841 pending_rounds.cc:62] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.410228 5841 pending_rounds.cc:68] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924838228856832 op_type: NO_OP noop_request { }
I20240507 08:12:21.410336 5841 raft_consensus.cc:2866] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.410434 5841 raft_consensus.cc:2258] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.411046 7748 raft_consensus.cc:504] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.411180 5841 ts_tablet_manager.cc:1875] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.411839 7748 leader_election.cc:288] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.412277 5656 consensus_peers.cc:485] T 5f0b399a2d074f4ca3626ff17bd5518e P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f0b399a2d074f4ca3626ff17bd5518e. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.412624 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3397e33b4cc7805a25e7535a383a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11"
I20240507 08:12:21.412806 5446 raft_consensus.cc:3037] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.413123 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6ba3397e33b4cc7805a25e7535a383a" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.414546 5654 leader_election.cc:341] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.414757 5843 tablet_bootstrap.cc:492] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.414958 5843 ts_tablet_manager.cc:1367] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20240507 08:12:21.415247 5446 raft_consensus.cc:2454] T b6ba3397e33b4cc7805a25e7535a383a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 1.
I20240507 08:12:21.415628 5656 leader_election.cc:302] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.416112 7748 raft_consensus.cc:2781] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.416007 5843 raft_consensus.cc:348] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.416321 7748 raft_consensus.cc:686] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Becoming Leader. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Running, Role: LEADER
I20240507 08:12:21.416326 5843 raft_consensus.cc:374] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.416404 5843 raft_consensus.cc:729] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.416630 7748 consensus_queue.cc:237] T b6ba3397e33b4cc7805a25e7535a383a P 1cbb887175f345ada2433adaec906fd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.416683 5843 consensus_queue.cc:260] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.417006 5840 tablet_bootstrap.cc:492] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.417147 5840 ts_tablet_manager.cc:1367] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240507 08:12:21.417459 5843 ts_tablet_manager.cc:1398] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.417771 5843 tablet_bootstrap.cc:492] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.417897 5840 raft_consensus.cc:348] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.418170 5840 raft_consensus.cc:374] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.418255 5840 raft_consensus.cc:729] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.418452 5840 consensus_queue.cc:260] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.419148 5840 ts_tablet_manager.cc:1398] T fccc127d380d4035905c1b152ab1017b P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:21.419432 5840 tablet_bootstrap.cc:492] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.419595 5843 tablet_bootstrap.cc:654] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.420930 5840 tablet_bootstrap.cc:654] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.430835 5840 tablet_bootstrap.cc:492] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.431022 5840 ts_tablet_manager.cc:1367] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20240507 08:12:21.431723 5840 raft_consensus.cc:348] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.431933 5840 raft_consensus.cc:374] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.432009 5840 raft_consensus.cc:729] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.432214 5840 consensus_queue.cc:260] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.432675 5840 ts_tablet_manager.cc:1398] T aa806508ec3e489a9b2c2fa9196198d5 P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:21.432999 5840 tablet_bootstrap.cc:492] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.434971 5840 tablet_bootstrap.cc:654] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.435873 5841 ts_tablet_manager.cc:1888] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.436532 5841 log.cc:1196] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/2b13fbddcab5460db69f43e63f490fba
I20240507 08:12:21.438192 5841 ts_tablet_manager.cc:1909] T 2b13fbddcab5460db69f43e63f490fba P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.438628 7720 raft_consensus.cc:482] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.438786 7720 raft_consensus.cc:504] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.438917 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1376) took 1653 ms. Trace:
I20240507 08:12:21.438999 5841 rpcz_store.cc:276] 0507 08:12:19.785840 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.786174 (+ 334us) service_pool.cc:225] Handling call
0507 08:12:21.409696 (+1623522us) ts_tablet_manager.cc:1120] Deleting tablet 2b13fbddcab5460db69f43e63f490fba
0507 08:12:21.409705 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.429149 (+ 19444us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.435765 (+ 6616us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.438882 (+ 3117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1531897}
I20240507 08:12:21.439357 7720 leader_election.cc:288] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.439399 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 2b13fbddcab5460db69f43e63f490fba (table default.Table_0eea053201704cd3a7492f74e4b3340e [id=fa95acb8c36d472295f9018cf37aaedd]) successfully deleted
W20240507 08:12:21.440443 5520 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:21.440441 5263 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_7c4a61a066d64809ada26672081b8ed7" } new_table_name: "default.Table_4ba369bc09f54797aa8d764f89486fb0" modify_external_catalogs: true
W20240507 08:12:21.441030 5517 leader_election.cc:341] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Not found: Tablet not found: 767905763ba14cb8bbc52151ae9f17e6
I20240507 08:12:21.441161 5517 leader_election.cc:302] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fe673739d8084e7da5d7edd722e857d8; no voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11
I20240507 08:12:21.441479 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1377) took 1622 ms. Trace:
I20240507 08:12:21.441567 5841 rpcz_store.cc:276] 0507 08:12:19.819093 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.819148 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.441344 (+1622196us) ts_tablet_manager.cc:1120] Deleting tablet 6e2759ebfa0840f8b21d749354768c8d
0507 08:12:21.441354 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.441442 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563208}
I20240507 08:12:21.441690 7720 raft_consensus.cc:2726] T 767905763ba14cb8bbc52151ae9f17e6 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20240507 08:12:21.441965 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 6e2759ebfa0840f8b21d749354768c8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2759ebfa0840f8b21d749354768c8d already in progress: creating tablet
I20240507 08:12:21.442884 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1378) took 1622 ms. Trace:
I20240507 08:12:21.443284 5841 rpcz_store.cc:276] 0507 08:12:19.820075 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.820237 (+ 162us) service_pool.cc:225] Handling call
0507 08:12:21.442768 (+1622531us) ts_tablet_manager.cc:1120] Deleting tablet 669ffac4b4f34551ba930f0a259e78a6
0507 08:12:21.442777 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.442852 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1564572}
W20240507 08:12:21.443809 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 669ffac4b4f34551ba930f0a259e78a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 669ffac4b4f34551ba930f0a259e78a6 already in progress: creating tablet
I20240507 08:12:21.444253 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1379) took 1623 ms. Trace:
I20240507 08:12:21.446133 5841 rpcz_store.cc:276] 0507 08:12:19.821040 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.821090 (+ 50us) service_pool.cc:225] Handling call
0507 08:12:21.444104 (+1623014us) ts_tablet_manager.cc:1120] Deleting tablet 4bd0f775bfa344b9b02afc875db46af7
0507 08:12:21.444116 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.444206 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1565856}
W20240507 08:12:21.446950 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 4bd0f775bfa344b9b02afc875db46af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd0f775bfa344b9b02afc875db46af7 already in progress: creating tablet
I20240507 08:12:21.447814 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1380) took 1624 ms. Trace:
I20240507 08:12:21.448860 5841 rpcz_store.cc:276] 0507 08:12:19.823607 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.823658 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:21.447702 (+1624044us) ts_tablet_manager.cc:1120] Deleting tablet 92e1e4cbc3e34af8bb56fc0ae41b7377
0507 08:12:21.447710 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.447781 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1569418}
W20240507 08:12:21.449455 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 already in progress: creating tablet
I20240507 08:12:21.449827 5841 tablet_replica.cc:330] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.451223 5841 raft_consensus.cc:2229] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.451754 5841 raft_consensus.cc:2258] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.452572 5841 ts_tablet_manager.cc:1875] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.455725 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54e3be7b44034d25ab402f11166e2ab5 consensus state has no leader
I20240507 08:12:21.459316 5259 catalog_manager.cc:5576] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443) does not have the latest schema for tablet 6f3bdb777b974524b5673a3f02883107 (table default.Table_cd3830ae1d3941de8d553267940b7b1a [id=98125a41d31245e6975dc9393d2faef7]). Expected version 1 got 0
W20240507 08:12:21.461477 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1652772fff2d4ed9b50bb3022623eaa8 consensus state has no leader
I20240507 08:12:21.464629 7201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap replayed 1/1 log segments. Stats: ops{read=823 overwritten=0 applied=823 ignored=0} inserts{seen=761 ignored=0} mutations{seen=957 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240507 08:12:21.465893 7201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae: Bootstrap complete.
I20240507 08:12:21.468183 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet b8c6f552832a45c4a2e28f40797db8fa with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:16 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.468685 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 14929efb0ed049d6b554598a5b8cb494 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:01 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.469853 5416 tablet_service.cc:1467] Processing CreateTablet for tablet b6a9017c700743ba933c62cf1bd178fc (DEFAULT_TABLE table=default.Table_2c24bc292e0e44fba3e26850b6615d06 [id=8034f800a2ed45858cc353fbef9921de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.470681 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 44f251110664402fac0e2abc77f11e8c (DEFAULT_TABLE table=default.Table_60d63759e9ab452b86714212d928185f [id=99954f02fa004e57970a32cbf375f276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.470940 5417 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab0edcef974d44828639bd18da444f (DEFAULT_TABLE table=default.Table_99fc6e6505b14088ab7e1344ee923296 [id=7cdf05aeaad34e32a14f0143ebb1663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.471767 5409 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706 (DEFAULT_TABLE table=default.Table_a20a31ad7635469c8353b7e7759c0a39 [id=dbd43dca22974cd484bdec67d2613727]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.473073 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b6a9017c700743ba933c62cf1bd178fc (DEFAULT_TABLE table=default.Table_2c24bc292e0e44fba3e26850b6615d06 [id=8034f800a2ed45858cc353fbef9921de]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.473199 5689 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab0edcef974d44828639bd18da444f (DEFAULT_TABLE table=default.Table_99fc6e6505b14088ab7e1344ee923296 [id=7cdf05aeaad34e32a14f0143ebb1663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.473440 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a9017c700743ba933c62cf1bd178fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.473624 5698 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac (DEFAULT_TABLE table=default.Table_f59032e865db4953a0e6c4517ae795ba [id=7d40b9c4a9d944d485026e160f1d386f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.473732 5685 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706 (DEFAULT_TABLE table=default.Table_a20a31ad7635469c8353b7e7759c0a39 [id=dbd43dca22974cd484bdec67d2613727]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.473906 5698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.474090 5685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.477807 5693 tablet_service.cc:1467] Processing CreateTablet for tablet 44f251110664402fac0e2abc77f11e8c (DEFAULT_TABLE table=default.Table_60d63759e9ab452b86714212d928185f [id=99954f02fa004e57970a32cbf375f276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.478192 5689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab0edcef974d44828639bd18da444f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.478148 5697 tablet_service.cc:1467] Processing CreateTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac (DEFAULT_TABLE table=default.Table_512dbcaed1c74258922227c3af764ede [id=47a7522e1159417796761e3ab737f7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.478771 5408 tablet_service.cc:1467] Processing CreateTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac (DEFAULT_TABLE table=default.Table_512dbcaed1c74258922227c3af764ede [id=47a7522e1159417796761e3ab737f7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.479250 5418 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac (DEFAULT_TABLE table=default.Table_f59032e865db4953a0e6c4517ae795ba [id=7d40b9c4a9d944d485026e160f1d386f]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.479648 5419 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc8a7342db4dd698d4e78e82500ef9 (DEFAULT_TABLE table=default.Table_015ef8e34abd4318ac03fb75c3e931a5 [id=59b0482d4357419d8a5fa7b9eed61e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.478796 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 32b73791d4ce4f25bfffcd55b640b6a4 (DEFAULT_TABLE table=default.Table_c6cb80106e7a43168975c570c122e5e2 [id=02e37cf61f294bf987e2654be6db4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.481009 5699 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc8a7342db4dd698d4e78e82500ef9 (DEFAULT_TABLE table=default.Table_015ef8e34abd4318ac03fb75c3e931a5 [id=59b0482d4357419d8a5fa7b9eed61e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.484009 5553 tablet_service.cc:1467] Processing CreateTablet for tablet 44f251110664402fac0e2abc77f11e8c (DEFAULT_TABLE table=default.Table_60d63759e9ab452b86714212d928185f [id=99954f02fa004e57970a32cbf375f276]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.484166 5559 tablet_service.cc:1467] Processing CreateTablet for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706 (DEFAULT_TABLE table=default.Table_a20a31ad7635469c8353b7e7759c0a39 [id=dbd43dca22974cd484bdec67d2613727]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.484412 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f251110664402fac0e2abc77f11e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.484572 5560 tablet_service.cc:1467] Processing CreateTablet for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac (DEFAULT_TABLE table=default.Table_512dbcaed1c74258922227c3af764ede [id=47a7522e1159417796761e3ab737f7f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.484961 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 39ab0edcef974d44828639bd18da444f (DEFAULT_TABLE table=default.Table_99fc6e6505b14088ab7e1344ee923296 [id=7cdf05aeaad34e32a14f0143ebb1663d]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.485111 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac (DEFAULT_TABLE table=default.Table_f59032e865db4953a0e6c4517ae795ba [id=7d40b9c4a9d944d485026e160f1d386f]), partition=RANGE (key) PARTITION UNBOUNDED
W20240507 08:12:21.485234 5378 consensus_peers.cc:485] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.485538 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 6dbc8a7342db4dd698d4e78e82500ef9 (DEFAULT_TABLE table=default.Table_015ef8e34abd4318ac03fb75c3e931a5 [id=59b0482d4357419d8a5fa7b9eed61e61]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.485935 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f251110664402fac0e2abc77f11e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.485982 5841 ts_tablet_manager.cc:1888] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240507 08:12:21.485989 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 32b73791d4ce4f25bfffcd55b640b6a4 (DEFAULT_TABLE table=default.Table_c6cb80106e7a43168975c570c122e5e2 [id=02e37cf61f294bf987e2654be6db4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.486089 5841 log.cc:1196] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/cbf7dd6413fb47108cce25cde80738fe
I20240507 08:12:21.486444 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc8a7342db4dd698d4e78e82500ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.486515 5841 ts_tablet_manager.cc:1909] T cbf7dd6413fb47108cce25cde80738fe P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.486924 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.487434 5417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab0edcef974d44828639bd18da444f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.487974 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.488444 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.488569 7762 raft_consensus.cc:482] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.488703 7762 raft_consensus.cc:504] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.488931 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b73791d4ce4f25bfffcd55b640b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.489370 7762 leader_election.cc:288] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.490067 5416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6a9017c700743ba933c62cf1bd178fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.490028 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2ed8516944a083c02e07045e44bd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:21.490527 5378 leader_election.cc:341] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.493501 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1381) took 1654 ms. Trace:
I20240507 08:12:21.493623 5841 rpcz_store.cc:276] 0507 08:12:19.838979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.839035 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:21.449761 (+1610726us) ts_tablet_manager.cc:1120] Deleting tablet cbf7dd6413fb47108cce25cde80738fe
0507 08:12:21.449769 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.462847 (+ 13078us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.485866 (+ 23019us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.493458 (+ 7592us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1571422}
I20240507 08:12:21.494004 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1382) took 1648 ms. Trace:
I20240507 08:12:21.494099 5841 rpcz_store.cc:276] 0507 08:12:19.845386 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.845440 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:21.493875 (+1648435us) ts_tablet_manager.cc:1120] Deleting tablet d0eed609cb24474592c5b66b63e7669f
0507 08:12:21.493884 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.493966 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615496}
I20240507 08:12:21.494228 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_1377728ccdeb402fa8508a16105e7f80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.494483 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1392) took 1562 ms. Trace:
W20240507 08:12:21.494683 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1377728ccdeb402fa8508a16105e7f80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.496150 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44f251110664402fac0e2abc77f11e8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.499374 6513 ts_tablet_manager.cc:1875] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.484689 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c1883eb03aa4f2b8fb65c29dbfbf706. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.504215 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b73791d4ce4f25bfffcd55b640b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.505795 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.508517 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3fa4e4f9e06d4b87a64e0163a3ecc7ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.510139 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39ab0edcef974d44828639bd18da444f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.510581 5843 tablet_bootstrap.cc:492] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.510697 5843 ts_tablet_manager.cc:1367] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.007s
I20240507 08:12:21.511320 5843 raft_consensus.cc:348] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.511487 5843 raft_consensus.cc:374] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.511529 5843 raft_consensus.cc:729] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.511744 5843 consensus_queue.cc:260] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.512323 5843 ts_tablet_manager.cc:1398] T 8a397c40278141b7bb80ea5fcdbb125b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240507 08:12:21.513257 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc8a7342db4dd698d4e78e82500ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.513958 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6dbc8a7342db4dd698d4e78e82500ef9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.517093 5843 tablet_bootstrap.cc:492] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.517764 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.518564 5843 tablet_bootstrap.cc:654] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.530717 7079 ts_tablet_manager.cc:1888] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.530473 7201 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:12:21.531925 7201 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39de2a69ed4492aab4d7a232b09c1ae, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.532626 7201 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 823, Last appended: 1.823, Last appended by leader: 823, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } }
I20240507 08:12:21.536880 5840 tablet_bootstrap.cc:492] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.537001 5840 ts_tablet_manager.cc:1367] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.104s user 0.012s sys 0.013s
I20240507 08:12:21.530941 7079 log.cc:1196] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/5471c74c33cd42e8aaf47f71f61adff7
I20240507 08:12:21.537895 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a91047a8a5414eff9c06ca9ca2b548e8" member_type: VOTER last_known_addr { host: "127.4.246.254" port: 37179 } } peers { permanent_uuid: "a39de2a69ed4492aab4d7a232b09c1ae" member_type: VOTER last_known_addr { host: "127.4.246.253" port: 43459 } } peers { permanent_uuid: "94c202eef7f3466fa3e703644b40879d" member_type: VOTER last_known_addr { host: "127.4.246.252" port: 45297 } } }
I20240507 08:12:21.538304 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: This master's current role is: FOLLOWER
I20240507 08:12:21.540169 7201 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39de2a69ed4492aab4d7a232b09c1ae [sys.catalog]: configured and running, proceeding with master startup.
I20240507 08:12:21.540593 5841 rpcz_store.cc:276] 0507 08:12:19.932169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.933655 (+ 1486us) service_pool.cc:225] Handling call
0507 08:12:21.494363 (+1560708us) ts_tablet_manager.cc:1120] Deleting tablet 5f9846e0eab74388af231e9942a6881a
0507 08:12:21.494372 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.494444 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1560570}
I20240507 08:12:21.540946 5841 tablet_replica.cc:330] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.541131 5841 raft_consensus.cc:2229] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.548736 5841 pending_rounds.cc:62] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.548816 5841 pending_rounds.cc:68] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924838471405568 op_type: NO_OP noop_request { }
I20240507 08:12:21.548939 5841 raft_consensus.cc:2866] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.549027 5841 raft_consensus.cc:2258] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.550616 5841 ts_tablet_manager.cc:1875] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.561018 5410 tablet_service.cc:1467] Processing CreateTablet for tablet fbfa7de41c354e39888028dd563eb36a (DEFAULT_TABLE table=default.Table_904d0fe58daf4a51a076b0b29aa4c942 [id=a3dd34fbf7734d5983d0a06ab4db9bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.561168 5422 tablet_service.cc:1467] Processing CreateTablet for tablet 89df9dcebed34d7da51b07b8e43880be (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.561452 5410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfa7de41c354e39888028dd563eb36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.561800 5414 tablet_service.cc:1467] Processing CreateTablet for tablet 048abf63fd164f22ac86129760e800f4 (DEFAULT_TABLE table=default.Table_7420d3e805d64c9096c3e5411ad789db [id=d5de7c77461c496fb4bfc1ce837857e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.562142 5427 tablet_service.cc:1467] Processing CreateTablet for tablet 2edcda81004c4e7880af58a6c3967515 (DEFAULT_TABLE table=default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8 [id=e14ffda35aca488b86925db1d94817c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.562461 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 85018a1ab8b84d579c5ccec679d2f5bf (DEFAULT_TABLE table=default.Table_06265a021c464ab0b3391f95db5c2354 [id=2b83dc70b37c438abda6b90f60aeaccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.562747 5413 tablet_service.cc:1467] Processing CreateTablet for tablet c6130da2ee3e4ecbbe13fcfae98a4953 (DEFAULT_TABLE table=default.Table_dffc373d1b8940d6bdcc3280f92d0743 [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.563026 5420 tablet_service.cc:1467] Processing CreateTablet for tablet ffd00e08c5444ec8bd65028431aea9ca (DEFAULT_TABLE table=default.Table_d608eb93e54b43d2b21ba5c735d0f778 [id=e73e80f76b134d67a3be0f2bbd8ff923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.561457 7762 raft_consensus.cc:482] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.563323 7762 raft_consensus.cc:504] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.563902 7762 leader_election.cc:288] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.564215 5412 tablet_service.cc:1467] Processing CreateTablet for tablet 35678404b67b4273b61fa5f922152fe3 (DEFAULT_TABLE table=default.Table_c4a924e93759482bba1dfca1998c326c [id=44278ac7fbfa43b7adbadc799c606d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.564616 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190baa48859640cf928642dccbebfceb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.564767 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet cbf7dd6413fb47108cce25cde80738fe (table default.Table_dffc373d1b8940d6bdcc3280f92d0743 [id=c59b446f65c842478dae29b4065fce5a]) successfully deleted
W20240507 08:12:21.565034 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet d0eed609cb24474592c5b66b63e7669f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d0eed609cb24474592c5b66b63e7669f already in progress: creating tablet
W20240507 08:12:21.565338 5378 leader_election.cc:341] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:21.565393 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5f9846e0eab74388af231e9942a6881a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5f9846e0eab74388af231e9942a6881a already in progress: creating tablet
I20240507 08:12:21.565711 5414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048abf63fd164f22ac86129760e800f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.569798 5427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edcda81004c4e7880af58a6c3967515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.573743 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85018a1ab8b84d579c5ccec679d2f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.576920 5841 ts_tablet_manager.cc:1888] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.577011 5841 log.cc:1196] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/a5976eca006448e0b9f2573acd738bb1
I20240507 08:12:21.577327 5841 ts_tablet_manager.cc:1909] T a5976eca006448e0b9f2573acd738bb1 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.577601 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1394) took 1645 ms. Trace:
I20240507 08:12:21.577744 5413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6130da2ee3e4ecbbe13fcfae98a4953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.578485 6513 ts_tablet_manager.cc:1888] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:21.578593 6513 log.cc:1196] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/43644b4405f44bfd9a1afcaedbd68db5
I20240507 08:12:21.578892 6513 ts_tablet_manager.cc:1909] T 43644b4405f44bfd9a1afcaedbd68db5 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.579489 7079 ts_tablet_manager.cc:1909] T 5471c74c33cd42e8aaf47f71f61adff7 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:21.577673 5841 rpcz_store.cc:276] 0507 08:12:19.932318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.932367 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:21.540893 (+1608526us) ts_tablet_manager.cc:1120] Deleting tablet a5976eca006448e0b9f2573acd738bb1
0507 08:12:21.540903 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.557799 (+ 16896us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.574394 (+ 16595us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.577567 (+ 3173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1579483}
I20240507 08:12:21.579752 5841 tablet_replica.cc:330] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.579838 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1307) took 2468 ms. Trace:
I20240507 08:12:21.579931 5841 raft_consensus.cc:2229] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.579938 7079 rpcz_store.cc:276] 0507 08:12:19.111167 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.111222 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.294800 (+2183578us) ts_tablet_manager.cc:1120] Deleting tablet 5471c74c33cd42e8aaf47f71f61adff7
0507 08:12:21.294814 (+ 14us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.352810 (+ 57996us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.530266 (+177456us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.579797 (+ 49531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183377}
I20240507 08:12:21.580209 5841 raft_consensus.cc:2258] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.580338 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1308) took 2437 ms. Trace:
I20240507 08:12:21.580427 7079 rpcz_store.cc:276] 0507 08:12:19.143302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.143357 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.580198 (+2436841us) ts_tablet_manager.cc:1120] Deleting tablet 01820ecad55a4097bc5917a8f7949fe7
0507 08:12:21.580206 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.580298 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2436645}
I20240507 08:12:21.580623 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet a5976eca006448e0b9f2573acd738bb1 (table default.Table_0ab8cb559bbe4154a11058e94d2ad349 [id=ade85b530c6f475592634b55a82394a3]) successfully deleted
I20240507 08:12:21.580802 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1309) took 2424 ms. Trace:
I20240507 08:12:21.580881 7079 rpcz_store.cc:276] 0507 08:12:19.156404 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.156459 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.580683 (+2424224us) ts_tablet_manager.cc:1120] Deleting tablet a905e345412541558b30c629a2eb86c5
0507 08:12:21.580691 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.580765 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2422795}
I20240507 08:12:21.581168 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1312) took 2382 ms. Trace:
I20240507 08:12:21.581240 7079 rpcz_store.cc:276] 0507 08:12:19.199018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.199071 (+ 53us) service_pool.cc:225] Handling call
0507 08:12:21.581070 (+2381999us) ts_tablet_manager.cc:1120] Deleting tablet aae2f13901a146b7a2019709b28930e9
0507 08:12:21.581078 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.581140 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381804}
I20240507 08:12:21.581781 5420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd00e08c5444ec8bd65028431aea9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.583667 5841 ts_tablet_manager.cc:1875] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.585372 5412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35678404b67b4273b61fa5f922152fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.588970 5841 ts_tablet_manager.cc:1888] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.589048 5841 log.cc:1196] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/43225934cef04109b3dbc057b230f032
I20240507 08:12:21.589334 5841 ts_tablet_manager.cc:1909] T 43225934cef04109b3dbc057b230f032 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.589814 5422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89df9dcebed34d7da51b07b8e43880be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.595861 5259 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2365998a0bb04c22bea71d6a205a57cf" } new_table_name: "default.Table_dd927f7f9ee0449e9efc4f42e76a71b8" modify_external_catalogs: true
I20240507 08:12:21.599248 5415 tablet_service.cc:1467] Processing CreateTablet for tablet ba28a6ecb3ce4e69811e44dcee0b02bb (DEFAULT_TABLE table=default.Table_d03405787efc42a89e644c26fe8f973e [id=6dc42310692445e18dd503d920dd533c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.599726 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28a6ecb3ce4e69811e44dcee0b02bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.601902 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1395) took 1640 ms. Trace:
I20240507 08:12:21.601987 5841 rpcz_store.cc:276] 0507 08:12:19.961400 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.961455 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.579712 (+1618257us) ts_tablet_manager.cc:1120] Deleting tablet 43225934cef04109b3dbc057b230f032
0507 08:12:21.579720 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.586348 (+ 6628us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.588907 (+ 2559us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.601879 (+ 12972us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1618060}
I20240507 08:12:21.602280 5841 tablet_replica.cc:330] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.602443 5841 raft_consensus.cc:2229] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.602581 5841 raft_consensus.cc:2258] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.602563 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 43225934cef04109b3dbc057b230f032 (table default.Table_0fafe65af5f34be4b333cb6baa327f94 [id=58282ffddcba4e4db7736a325a4a530a]) successfully deleted
I20240507 08:12:21.603067 5841 ts_tablet_manager.cc:1875] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.606395 5840 raft_consensus.cc:348] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.606607 5840 raft_consensus.cc:374] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.606664 5840 raft_consensus.cc:729] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.606834 5840 consensus_queue.cc:260] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.607961 7743 raft_consensus.cc:482] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.608103 7743 raft_consensus.cc:504] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.608369 5841 ts_tablet_manager.cc:1888] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.608450 5841 log.cc:1196] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/156bf04b57724134a45fa9fc477596c5
I20240507 08:12:21.608702 5841 ts_tablet_manager.cc:1909] T 156bf04b57724134a45fa9fc477596c5 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.608788 7743 leader_election.cc:288] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.609263 5697 tablet_service.cc:1467] Processing CreateTablet for tablet 32b73791d4ce4f25bfffcd55b640b6a4 (DEFAULT_TABLE table=default.Table_c6cb80106e7a43168975c570c122e5e2 [id=02e37cf61f294bf987e2654be6db4e0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.609640 5697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32b73791d4ce4f25bfffcd55b640b6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.609777 5682 tablet_service.cc:1467] Processing CreateTablet for tablet 89df9dcebed34d7da51b07b8e43880be (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.610057 5682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89df9dcebed34d7da51b07b8e43880be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.610175 5695 tablet_service.cc:1467] Processing CreateTablet for tablet c6130da2ee3e4ecbbe13fcfae98a4953 (DEFAULT_TABLE table=default.Table_dffc373d1b8940d6bdcc3280f92d0743 [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.610502 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6130da2ee3e4ecbbe13fcfae98a4953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.613792 5692 tablet_service.cc:1467] Processing CreateTablet for tablet 048abf63fd164f22ac86129760e800f4 (DEFAULT_TABLE table=default.Table_7420d3e805d64c9096c3e5411ad789db [id=d5de7c77461c496fb4bfc1ce837857e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.613960 5687 tablet_service.cc:1467] Processing CreateTablet for tablet fbfa7de41c354e39888028dd563eb36a (DEFAULT_TABLE table=default.Table_904d0fe58daf4a51a076b0b29aa4c942 [id=a3dd34fbf7734d5983d0a06ab4db9bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.614104 5692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048abf63fd164f22ac86129760e800f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.614223 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfa7de41c354e39888028dd563eb36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.615219 5717 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2ed8516944a083c02e07045e44bd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.615253 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190baa48859640cf928642dccbebfceb" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.615911 5717 raft_consensus.cc:2454] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
W20240507 08:12:21.617331 5377 leader_election.cc:341] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.617440 5377 leader_election.cc:302] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.617799 5693 tablet_service.cc:1467] Processing CreateTablet for tablet ba28a6ecb3ce4e69811e44dcee0b02bb (DEFAULT_TABLE table=default.Table_d03405787efc42a89e644c26fe8f973e [id=6dc42310692445e18dd503d920dd533c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618098 5700 tablet_service.cc:1467] Processing CreateTablet for tablet c844ad4d1b42444f979e5e65a6f41a5c (DEFAULT_TABLE table=default.Table_1b17db44e5fc453daefcada73f3e7f24 [id=bbc15db0be1d4e64bd934feefd204658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618273 5690 tablet_service.cc:1467] Processing CreateTablet for tablet 35678404b67b4273b61fa5f922152fe3 (DEFAULT_TABLE table=default.Table_c4a924e93759482bba1dfca1998c326c [id=44278ac7fbfa43b7adbadc799c606d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618518 5688 tablet_service.cc:1467] Processing CreateTablet for tablet 2af82df3db9644b2ae8ff48a15c962a1 (DEFAULT_TABLE table=default.Table_56f7efcd33244fe0b455eabef3d06f68 [id=58fb222f5ee44fd1bbc85cd483caa333]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618662 5691 tablet_service.cc:1467] Processing CreateTablet for tablet 2edcda81004c4e7880af58a6c3967515 (DEFAULT_TABLE table=default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8 [id=e14ffda35aca488b86925db1d94817c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618101 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28a6ecb3ce4e69811e44dcee0b02bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.618924 5684 tablet_service.cc:1467] Processing CreateTablet for tablet d0d15ec7033c4647b10e4085a097eebb (DEFAULT_TABLE table=default.Table_baf4dd7a9fef47a08fbd55804100dba3 [id=a416875fe0c44596b2ab98d34a1e5e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.618880 5408 tablet_service.cc:1467] Processing CreateTablet for tablet d0d15ec7033c4647b10e4085a097eebb (DEFAULT_TABLE table=default.Table_baf4dd7a9fef47a08fbd55804100dba3 [id=a416875fe0c44596b2ab98d34a1e5e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.619191 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d15ec7033c4647b10e4085a097eebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.619282 5408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d15ec7033c4647b10e4085a097eebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.620456 5423 tablet_service.cc:1467] Processing CreateTablet for tablet 2af82df3db9644b2ae8ff48a15c962a1 (DEFAULT_TABLE table=default.Table_56f7efcd33244fe0b455eabef3d06f68 [id=58fb222f5ee44fd1bbc85cd483caa333]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.620741 5423 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af82df3db9644b2ae8ff48a15c962a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.621362 7649 raft_consensus.cc:482] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.621475 7649 raft_consensus.cc:504] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.613792 5699 tablet_service.cc:1467] Processing CreateTablet for tablet df4dbdedb0664f6c8f242c1dba020dbd (DEFAULT_TABLE table=default.Table_1a1a5e38eff2435c9be461ea3ed65ca8 [id=c4b9c87946be4388b161f5a2232c5e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.617834 5681 tablet_service.cc:1467] Processing CreateTablet for tablet 85018a1ab8b84d579c5ccec679d2f5bf (DEFAULT_TABLE table=default.Table_06265a021c464ab0b3391f95db5c2354 [id=2b83dc70b37c438abda6b90f60aeaccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.621702 5683 tablet_service.cc:1467] Processing CreateTablet for tablet ffd00e08c5444ec8bd65028431aea9ca (DEFAULT_TABLE table=default.Table_d608eb93e54b43d2b21ba5c735d0f778 [id=e73e80f76b134d67a3be0f2bbd8ff923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.622002 7649 leader_election.cc:288] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.622642 5409 tablet_service.cc:1467] Processing CreateTablet for tablet c844ad4d1b42444f979e5e65a6f41a5c (DEFAULT_TABLE table=default.Table_1b17db44e5fc453daefcada73f3e7f24 [id=bbc15db0be1d4e64bd934feefd204658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.622931 5409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c844ad4d1b42444f979e5e65a6f41a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.621722 5691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edcda81004c4e7880af58a6c3967515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.624886 5690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35678404b67b4273b61fa5f922152fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.625031 7722 consensus_queue.cc:1035] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.625178 5418 tablet_service.cc:1467] Processing CreateTablet for tablet df4dbdedb0664f6c8f242c1dba020dbd (DEFAULT_TABLE table=default.Table_1a1a5e38eff2435c9be461ea3ed65ca8 [id=c4b9c87946be4388b161f5a2232c5e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.625422 7722 consensus_queue.cc:1035] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240507 08:12:21.625761 7722 raft_consensus.cc:2726] T 190baa48859640cf928642dccbebfceb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.625944 5688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af82df3db9644b2ae8ff48a15c962a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.626312 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c844ad4d1b42444f979e5e65a6f41a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.626345 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet 5471c74c33cd42e8aaf47f71f61adff7 (table default.Table_e85104a6436841ad8d9510f2e54233c3 [id=1cbcaf54182340998beb227653bb24ae]) successfully deleted
I20240507 08:12:21.626893 5418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4dbdedb0664f6c8f242c1dba020dbd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.628552 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 01820ecad55a4097bc5917a8f7949fe7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01820ecad55a4097bc5917a8f7949fe7 already in progress: creating tablet
I20240507 08:12:21.629719 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85018a1ab8b84d579c5ccec679d2f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.630219 5699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4dbdedb0664f6c8f242c1dba020dbd. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.630335 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet a905e345412541558b30c629a2eb86c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a905e345412541558b30c629a2eb86c5 already in progress: creating tablet
W20240507 08:12:21.630558 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet aae2f13901a146b7a2019709b28930e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aae2f13901a146b7a2019709b28930e9 already in progress: creating tablet
I20240507 08:12:21.630672 5683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd00e08c5444ec8bd65028431aea9ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.631567 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1242) took 5124 ms. Trace:
I20240507 08:12:21.631651 6513 rpcz_store.cc:276] 0507 08:12:16.507478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.507925 (+ 447us) service_pool.cc:225] Handling call
0507 08:12:21.323866 (+4815941us) ts_tablet_manager.cc:1120] Deleting tablet 43644b4405f44bfd9a1afcaedbd68db5
0507 08:12:21.323874 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.531377 (+207503us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.578390 (+ 47013us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.631550 (+ 53160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":51811,"tablet-delete.queue_time_us":4815706}
I20240507 08:12:21.632683 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 43644b4405f44bfd9a1afcaedbd68db5 (table default.Table_e9fbf7d3226c4da89aa7ca9c2e502d5a [id=9a07ac74f53140c18c5ee38d4377484d]) successfully deleted
W20240507 08:12:21.641830 5256 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4a1400b30cbe4411bce4addb62c8c979 has no consensus state
I20240507 08:12:21.641906 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbc152c3898a4dbcbaa844b31b5598cb" } modify_external_catalogs: true
I20240507 08:12:21.642056 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_bbc152c3898a4dbcbaa844b31b5598cb" } modify_external_catalogs: true
I20240507 08:12:21.642190 5256 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_21078c5be70644d5934a682d5e65d7f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.642688 5256 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21078c5be70644d5934a682d5e65d7f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.643749 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b65e251f3d5c4414afe413a54a519d85
I20240507 08:12:21.644312 5258 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2311302968aa441c8cfc10afe6e08044" } modify_external_catalogs: true
I20240507 08:12:21.644457 5258 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2311302968aa441c8cfc10afe6e08044" } modify_external_catalogs: true
I20240507 08:12:21.646631 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb2520f3b7b74ba79f88fa52ed0ff8c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.647073 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2520f3b7b74ba79f88fa52ed0ff8c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.647776 5695 tablet_service.cc:1514] Processing DeleteTablet for tablet b65e251f3d5c4414afe413a54a519d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.648335 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6fd5e754c95b480a8ecd9e7f593b7260" } modify_external_catalogs: true
I20240507 08:12:21.648480 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_6fd5e754c95b480a8ecd9e7f593b7260" } modify_external_catalogs: true
I20240507 08:12:21.650491 5262 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8" } modify_external_catalogs: true
I20240507 08:12:21.650633 5262 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8" } modify_external_catalogs: true
W20240507 08:12:21.652197 5263 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2adde63d55e46158a80be0c16667963 consensus state has no leader
I20240507 08:12:21.652508 5263 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e3d2024ebd504b9997686d0fe4c244ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.652904 5263 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3d2024ebd504b9997686d0fe4c244ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.654145 7762 raft_consensus.cc:482] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.654312 7762 raft_consensus.cc:504] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.655076 7762 leader_election.cc:288] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.660915 5419 tablet_service.cc:1514] Processing DeleteTablet for tablet b65e251f3d5c4414afe413a54a519d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.661484 7762 raft_consensus.cc:482] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.661644 7762 raft_consensus.cc:504] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.662379 7762 leader_election.cc:288] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.667068 5695 tablet_service.cc:1467] Processing CreateTablet for tablet f7b880ca1d444e8889207ff486edb044 (DEFAULT_TABLE table=default.Table_0e29c9c70e364d598433f81aa779ae4e [id=0a037a998e714bbd8b6c1a8bba9c9878]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.668381 5687 tablet_service.cc:1467] Processing CreateTablet for tablet b0adca43a2fc432393c62506206374bd (DEFAULT_TABLE table=default.Table_f980c1cb998f49dea75f044a4201d5c4 [id=4926c528b839415b9142e5b9b6449a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.669337 5693 tablet_service.cc:1467] Processing CreateTablet for tablet fe6cce5ec0a74b6987e4466e1b7d964a (DEFAULT_TABLE table=default.Table_76d0ef5741d34f578a0810ddcde0bc44 [id=a5cd368e803248a7be2b08699fd51dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.674131 5411 tablet_service.cc:1467] Processing CreateTablet for tablet b0adca43a2fc432393c62506206374bd (DEFAULT_TABLE table=default.Table_f980c1cb998f49dea75f044a4201d5c4 [id=4926c528b839415b9142e5b9b6449a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.674535 5411 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0adca43a2fc432393c62506206374bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.678236 5424 tablet_service.cc:1467] Processing CreateTablet for tablet fe6cce5ec0a74b6987e4466e1b7d964a (DEFAULT_TABLE table=default.Table_76d0ef5741d34f578a0810ddcde0bc44 [id=a5cd368e803248a7be2b08699fd51dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.678613 5424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6cce5ec0a74b6987e4466e1b7d964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.690083 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1393) took 1757 ms. Trace:
I20240507 08:12:21.690179 5841 rpcz_store.cc:276] 0507 08:12:19.932245 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.933826 (+ 1581us) service_pool.cc:225] Handling call
0507 08:12:21.602240 (+1668414us) ts_tablet_manager.cc:1120] Deleting tablet 156bf04b57724134a45fa9fc477596c5
0507 08:12:21.602248 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.605755 (+ 3507us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.608303 (+ 2548us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.690066 (+ 81763us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":81081,"tablet-delete.queue_time_us":1640382}
I20240507 08:12:21.690475 5841 tablet_replica.cc:330] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.690666 5841 raft_consensus.cc:2229] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.690752 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 156bf04b57724134a45fa9fc477596c5 (table default.Table_3ffbb2e35cc34f288580fefe1e79ebfd [id=004c644ce7ae421f948ccf0fdef44224]) successfully deleted
I20240507 08:12:21.690929 5841 pending_rounds.cc:62] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.690987 5841 pending_rounds.cc:68] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924838419406848 op_type: NO_OP noop_request { }
I20240507 08:12:21.691092 5841 raft_consensus.cc:2866] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.691175 5841 raft_consensus.cc:2258] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.691670 5419 tablet_service.cc:1467] Processing CreateTablet for tablet f7b880ca1d444e8889207ff486edb044 (DEFAULT_TABLE table=default.Table_0e29c9c70e364d598433f81aa779ae4e [id=0a037a998e714bbd8b6c1a8bba9c9878]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.691735 5841 ts_tablet_manager.cc:1875] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.692548 5419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b880ca1d444e8889207ff486edb044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.695099 5552 tablet_service.cc:1467] Processing CreateTablet for tablet ba28a6ecb3ce4e69811e44dcee0b02bb (DEFAULT_TABLE table=default.Table_d03405787efc42a89e644c26fe8f973e [id=6dc42310692445e18dd503d920dd533c]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.695168 5556 tablet_service.cc:1467] Processing CreateTablet for tablet 35678404b67b4273b61fa5f922152fe3 (DEFAULT_TABLE table=default.Table_c4a924e93759482bba1dfca1998c326c [id=44278ac7fbfa43b7adbadc799c606d50]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.695931 5553 tablet_service.cc:1467] Processing CreateTablet for tablet fbfa7de41c354e39888028dd563eb36a (DEFAULT_TABLE table=default.Table_904d0fe58daf4a51a076b0b29aa4c942 [id=a3dd34fbf7734d5983d0a06ab4db9bd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.697412 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43a80d38ed4788a13421b245fe6e8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.697722 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba28a6ecb3ce4e69811e44dcee0b02bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.698827 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1313) took 2494 ms. Trace:
W20240507 08:12:21.699600 5654 consensus_peers.cc:485] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:21.699882 5654 consensus_peers.cc:485] T 3ccc3e67eeb34c42b36edde7bacd23a5 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:21.700174 5654 leader_election.cc:341] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.700151 5548 tablet_service.cc:1514] Processing DeleteTablet for tablet 242ea35c61404a91b1211c5ad9207728 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:06 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.700492 5841 ts_tablet_manager.cc:1888] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.700644 5841 log.cc:1196] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/63ccdad494794192b3c37c8d5cbfcb17
I20240507 08:12:21.700707 5558 tablet_service.cc:1467] Processing CreateTablet for tablet ffd00e08c5444ec8bd65028431aea9ca (DEFAULT_TABLE table=default.Table_d608eb93e54b43d2b21ba5c735d0f778 [id=e73e80f76b134d67a3be0f2bbd8ff923]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.701081 5841 ts_tablet_manager.cc:1909] T 63ccdad494794192b3c37c8d5cbfcb17 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.701428 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1397) took 1737 ms. Trace:
I20240507 08:12:21.701594 5841 rpcz_store.cc:276] 0507 08:12:19.964449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.964522 (+ 73us) service_pool.cc:225] Handling call
0507 08:12:21.690438 (+1725916us) ts_tablet_manager.cc:1120] Deleting tablet 63ccdad494794192b3c37c8d5cbfcb17
0507 08:12:21.690445 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.696703 (+ 6258us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.700417 (+ 3714us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.701406 (+ 989us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1725776}
I20240507 08:12:21.702180 5841 tablet_replica.cc:330] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.702325 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 63ccdad494794192b3c37c8d5cbfcb17 (table default.Table_c2d5a28991f9466486b34132cbd61131 [id=11df3920eb394624a4d2e85471cba9d9]) successfully deleted
I20240507 08:12:21.702466 5841 raft_consensus.cc:2229] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:21.703054 5654 consensus_peers.cc:485] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.703115 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35678404b67b4273b61fa5f922152fe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.703315 5841 raft_consensus.cc:2258] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.704495 5841 ts_tablet_manager.cc:1875] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.705904 7079 rpcz_store.cc:276] 0507 08:12:19.199164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.199245 (+ 81us) service_pool.cc:225] Handling call
0507 08:12:21.581406 (+2382161us) ts_tablet_manager.cc:1120] Deleting tablet c1935dd9e2a34c49bb3e0021f46bc765
0507 08:12:21.581413 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.693594 (+112181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2381878}
I20240507 08:12:21.706350 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005100f64fd49a3ac29d34aa728fdfb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.705753 7820 raft_consensus.cc:482] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.706845 5563 tablet_service.cc:1467] Processing CreateTablet for tablet 89df9dcebed34d7da51b07b8e43880be (DEFAULT_TABLE table=default.Table_e43fa98024c444e481ad778a98287def [id=ad1b4ae1eb3449a4bb0b6ec8d8c25554]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.707485 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fbfa7de41c354e39888028dd563eb36a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.709714 5558 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffd00e08c5444ec8bd65028431aea9ca. 1 dirs total, 0 dirs full, 0 dirs failed
W20240507 08:12:21.713439 5251 catalog_manager.cc:4628] Async tablet task AlterTable RPC for tablet e54273ed92f640d98b853a7661ae3e26 (table default.Table_82e54470d4fa4afc8451693317fc1d4e [id=362c05703392453ebeef62e6937a9139], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e54273ed92f640d98b853a7661ae3e26 consensus state has no leader
I20240507 08:12:21.714272 5559 tablet_service.cc:1467] Processing CreateTablet for tablet df4dbdedb0664f6c8f242c1dba020dbd (DEFAULT_TABLE table=default.Table_1a1a5e38eff2435c9be461ea3ed65ca8 [id=c4b9c87946be4388b161f5a2232c5e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.715657 5841 ts_tablet_manager.cc:1888] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.715751 5841 log.cc:1196] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d17b6a2f4aa14f2a927fa9ad56a62a5a
I20240507 08:12:21.716044 5841 ts_tablet_manager.cc:1909] T d17b6a2f4aa14f2a927fa9ad56a62a5a P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.716285 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1396) took 1754 ms. Trace:
I20240507 08:12:21.716346 5841 rpcz_store.cc:276] 0507 08:12:19.961532 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.962774 (+ 1242us) service_pool.cc:225] Handling call
0507 08:12:21.702128 (+1739354us) ts_tablet_manager.cc:1120] Deleting tablet d17b6a2f4aa14f2a927fa9ad56a62a5a
0507 08:12:21.702137 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.712881 (+ 10744us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.715584 (+ 2703us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.716270 (+ 686us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1727244}
I20240507 08:12:21.716758 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d17b6a2f4aa14f2a927fa9ad56a62a5a (table default.Table_28dc4a4ae86d46e79217cc266d671289 [id=0f762f862f334dcdaf39bad447d697c5]) successfully deleted
I20240507 08:12:21.718003 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1246) took 5156 ms. Trace:
I20240507 08:12:21.718093 6513 rpcz_store.cc:276] 0507 08:12:16.561983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.562943 (+ 960us) service_pool.cc:225] Handling call
0507 08:12:21.631942 (+5068999us) ts_tablet_manager.cc:1120] Deleting tablet 310c601abead42e0bd2a2df1a0e16ec2
0507 08:12:21.631950 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.717982 (+ 86032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5068830}
I20240507 08:12:21.718537 5693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6cce5ec0a74b6987e4466e1b7d964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.719260 5562 tablet_service.cc:1467] Processing CreateTablet for tablet 2edcda81004c4e7880af58a6c3967515 (DEFAULT_TABLE table=default.Table_3863be8dc4a04cd7bac5c6e876aa6ca8 [id=e14ffda35aca488b86925db1d94817c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.723299 5695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b880ca1d444e8889207ff486edb044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.723524 5687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0adca43a2fc432393c62506206374bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.724193 5563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89df9dcebed34d7da51b07b8e43880be. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.724965 5553 tablet_service.cc:1467] Processing CreateTablet for tablet c844ad4d1b42444f979e5e65a6f41a5c (DEFAULT_TABLE table=default.Table_1b17db44e5fc453daefcada73f3e7f24 [id=bbc15db0be1d4e64bd934feefd204658]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.725025 5556 tablet_service.cc:1467] Processing CreateTablet for tablet c6130da2ee3e4ecbbe13fcfae98a4953 (DEFAULT_TABLE table=default.Table_dffc373d1b8940d6bdcc3280f92d0743 [id=c59b446f65c842478dae29b4065fce5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.723836 7820 raft_consensus.cc:504] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.726862 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1403) took 1542 ms. Trace:
I20240507 08:12:21.726979 5548 tablet_service.cc:1467] Processing CreateTablet for tablet 048abf63fd164f22ac86129760e800f4 (DEFAULT_TABLE table=default.Table_7420d3e805d64c9096c3e5411ad789db [id=d5de7c77461c496fb4bfc1ce837857e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.726964 5841 rpcz_store.cc:276] 0507 08:12:20.184193 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.184380 (+ 187us) service_pool.cc:225] Handling call
0507 08:12:21.726729 (+1542349us) ts_tablet_manager.cc:1120] Deleting tablet c42f6e958dd94e3cbecd5c4bd6a6d829
0507 08:12:21.726741 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.726846 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542186}
W20240507 08:12:21.727363 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c42f6e958dd94e3cbecd5c4bd6a6d829 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c42f6e958dd94e3cbecd5c4bd6a6d829 already in progress: creating tablet
I20240507 08:12:21.727679 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2edcda81004c4e7880af58a6c3967515. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.727892 6513 tablet_replica.cc:330] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.728715 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1404) took 1544 ms. Trace:
I20240507 08:12:21.728796 5841 rpcz_store.cc:276] 0507 08:12:20.184292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.184678 (+ 386us) service_pool.cc:225] Handling call
0507 08:12:21.728619 (+1543941us) ts_tablet_manager.cc:1120] Deleting tablet 3f30af566be44760828e1672d83fc903
0507 08:12:21.728628 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.728700 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540710}
W20240507 08:12:21.729136 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 3f30af566be44760828e1672d83fc903 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3f30af566be44760828e1672d83fc903 already in progress: creating tablet
I20240507 08:12:21.729671 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6130da2ee3e4ecbbe13fcfae98a4953. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.730310 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1402) took 1546 ms. Trace:
I20240507 08:12:21.730383 5841 rpcz_store.cc:276] 0507 08:12:20.184041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.184575 (+ 534us) service_pool.cc:225] Handling call
0507 08:12:21.730219 (+1545644us) ts_tablet_manager.cc:1120] Deleting tablet 5370d022aa904900a28842b887b13187
0507 08:12:21.730228 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.730296 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1541907}
W20240507 08:12:21.730719 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5370d022aa904900a28842b887b13187 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5370d022aa904900a28842b887b13187 already in progress: creating tablet
I20240507 08:12:21.731967 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1405) took 1513 ms. Trace:
I20240507 08:12:21.732481 5841 rpcz_store.cc:276] 0507 08:12:20.218391 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.218454 (+ 63us) service_pool.cc:225] Handling call
0507 08:12:21.731874 (+1513420us) ts_tablet_manager.cc:1120] Deleting tablet 9354ceaf6f6d4f529d9dbe7b78360b82
0507 08:12:21.731882 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.731953 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513212}
W20240507 08:12:21.735972 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 9354ceaf6f6d4f529d9dbe7b78360b82 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9354ceaf6f6d4f529d9dbe7b78360b82 already in progress: creating tablet
I20240507 08:12:21.736418 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1406) took 1506 ms. Trace:
W20240507 08:12:21.736575 5520 leader_election.cc:341] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 53bc9bf1c04e4015a152cb750184c9ce
I20240507 08:12:21.737198 5550 tablet_service.cc:1514] Processing DeleteTablet for tablet b65e251f3d5c4414afe413a54a519d85 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.737282 5841 rpcz_store.cc:276] 0507 08:12:20.229778 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.230139 (+ 361us) service_pool.cc:225] Handling call
0507 08:12:21.736314 (+1506175us) ts_tablet_manager.cc:1120] Deleting tablet 401402afc04f4a9082c02ca5bf39d6a1
0507 08:12:21.736326 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.736404 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506014}
I20240507 08:12:21.737557 7720 raft_consensus.cc:482] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:21.737735 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 401402afc04f4a9082c02ca5bf39d6a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 401402afc04f4a9082c02ca5bf39d6a1 already in progress: creating tablet
I20240507 08:12:21.737732 7720 raft_consensus.cc:504] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.737545 5549 tablet_service.cc:1467] Processing CreateTablet for tablet 85018a1ab8b84d579c5ccec679d2f5bf (DEFAULT_TABLE table=default.Table_06265a021c464ab0b3391f95db5c2354 [id=2b83dc70b37c438abda6b90f60aeaccc]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.738379 5841 tablet_replica.cc:330] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.738497 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1319) took 2507 ms. Trace:
I20240507 08:12:21.739228 5841 raft_consensus.cc:2229] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.739460 5841 pending_rounds.cc:62] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.739521 5841 pending_rounds.cc:68] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924839454597120 op_type: NO_OP noop_request { }
I20240507 08:12:21.739385 7079 rpcz_store.cc:276] 0507 08:12:19.216823 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.216874 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:21.706346 (+2489472us) ts_tablet_manager.cc:1120] Deleting tablet d3f120d9fbf542d39508903883ecb5e6
0507 08:12:21.706381 (+ 35us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.724643 (+ 18262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2489285}
I20240507 08:12:21.739634 5841 raft_consensus.cc:2866] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.740715 5841 raft_consensus.cc:2258] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.741055 7720 leader_election.cc:288] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.741801 5520 leader_election.cc:341] T 242ea35c61404a91b1211c5ad9207728 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 242ea35c61404a91b1211c5ad9207728
I20240507 08:12:21.741964 5562 tablet_service.cc:1467] Processing CreateTablet for tablet f7b880ca1d444e8889207ff486edb044 (DEFAULT_TABLE table=default.Table_0e29c9c70e364d598433f81aa779ae4e [id=0a037a998e714bbd8b6c1a8bba9c9878]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.742063 5552 tablet_service.cc:1467] Processing CreateTablet for tablet 2af82df3db9644b2ae8ff48a15c962a1 (DEFAULT_TABLE table=default.Table_56f7efcd33244fe0b455eabef3d06f68 [id=58fb222f5ee44fd1bbc85cd483caa333]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.742187 5841 ts_tablet_manager.cc:1875] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.742292 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1322) took 2307 ms. Trace:
I20240507 08:12:21.742364 7079 rpcz_store.cc:276] 0507 08:12:19.435189 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.435546 (+ 357us) service_pool.cc:225] Handling call
0507 08:12:21.739769 (+2304223us) ts_tablet_manager.cc:1120] Deleting tablet ece0e5eae0d949149dd3bd3f4bbd4ca7
0507 08:12:21.739779 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.742278 (+ 2499us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2304039}
I20240507 08:12:21.742933 5559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet df4dbdedb0664f6c8f242c1dba020dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.743737 5552 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2af82df3db9644b2ae8ff48a15c962a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.743927 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1323) took 2244 ms. Trace:
I20240507 08:12:21.746605 5550 tablet_service.cc:1467] Processing CreateTablet for tablet b0adca43a2fc432393c62506206374bd (DEFAULT_TABLE table=default.Table_f980c1cb998f49dea75f044a4201d5c4 [id=4926c528b839415b9142e5b9b6449a63]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.755990 5841 ts_tablet_manager.cc:1888] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.756103 5841 log.cc:1196] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/b31fdadee60e4f5e94e92c536e18774d
I20240507 08:12:21.756464 5841 ts_tablet_manager.cc:1909] T b31fdadee60e4f5e94e92c536e18774d P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.758405 5257 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f
I20240507 08:12:21.759167 5257 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_7a8f28df22594060a8122e821264ca5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.759358 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:21.759675 5257 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a8f28df22594060a8122e821264ca5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.760641 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 8d6b0ccf73cc46a3a7815e5d09ed7614
I20240507 08:12:21.760733 5264 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: default.Table_18cce6f7f33d4b8e92f4c21a57a93eae [id=bcb7780193444ef29bbd00efd71c56a7] alter complete (version 2)
I20240507 08:12:21.761104 5260 master_service.cc:946] ReplaceTablet: received request to replace tablet 1227b63fbc6b4625bef0b37404b76a6f from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:21.761307 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a12a3159deed4b559e3ffa8d655dfd65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.762253 5414 tablet_service.cc:1514] Processing DeleteTablet for tablet 190baa48859640cf928642dccbebfceb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:11:58 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.762253 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.762607 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_9507adbda5534cbea160a6954098d837"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20240507 08:12:21.762722 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_f3940a8999cb4633b1d82922022afe2f" } new_table_name: "default.Table_732491a0ff8b4cccb4518dc1858dd03e" modify_external_catalogs: true
W20240507 08:12:21.763015 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9507adbda5534cbea160a6954098d837 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.763957 5258 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet b069d601b8fb4e75934675e7996a683a
I20240507 08:12:21.764510 5258 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_2468219608d6479fba6a3ef7216bd1a0" } new_table_name: "default.Table_01374c79efc54c2393943fdd63d53474" modify_external_catalogs: true
W20240507 08:12:21.765342 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a12a3159deed4b559e3ffa8d655dfd65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.765870 5256 master_service.cc:946] ReplaceTablet: received request to replace tablet 665b3736350a4121ac62014a81e5669d from {username='slave'} at 127.0.0.1:44738
I20240507 08:12:21.768863 7079 rpcz_store.cc:276] 0507 08:12:19.499715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.499770 (+ 55us) service_pool.cc:225] Handling call
0507 08:12:21.742566 (+2242796us) ts_tablet_manager.cc:1120] Deleting tablet e797fee2c43c4461a5cb9e2f0a2bb0fc
0507 08:12:21.742574 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.743911 (+ 1337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2242612}
W20240507 08:12:21.769760 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet c1935dd9e2a34c49bb3e0021f46bc765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1935dd9e2a34c49bb3e0021f46bc765 already in progress: creating tablet
I20240507 08:12:21.771679 5410 tablet_service.cc:1514] Processing DeleteTablet for tablet b069d601b8fb4e75934675e7996a683a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:21.775153 5555 tablet_service.cc:1467] Processing CreateTablet for tablet fe6cce5ec0a74b6987e4466e1b7d964a (DEFAULT_TABLE table=default.Table_76d0ef5741d34f578a0810ddcde0bc44 [id=a5cd368e803248a7be2b08699fd51dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.777079 5548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 048abf63fd164f22ac86129760e800f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.777395 5560 tablet_service.cc:1467] Processing CreateTablet for tablet d0d15ec7033c4647b10e4085a097eebb (DEFAULT_TABLE table=default.Table_baf4dd7a9fef47a08fbd55804100dba3 [id=a416875fe0c44596b2ab98d34a1e5e97]), partition=RANGE (key) PARTITION UNBOUNDED
I20240507 08:12:21.777406 5550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b0adca43a2fc432393c62506206374bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.778390 5553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c844ad4d1b42444f979e5e65a6f41a5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.780393 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1407) took 1474 ms. Trace:
I20240507 08:12:21.780505 5841 rpcz_store.cc:276] 0507 08:12:20.305499 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.323633 (+ 18134us) service_pool.cc:225] Handling call
0507 08:12:21.738337 (+1414704us) ts_tablet_manager.cc:1120] Deleting tablet b31fdadee60e4f5e94e92c536e18774d
0507 08:12:21.738347 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.746819 (+ 8472us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.755910 (+ 9091us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.780372 (+ 24462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1414440}
I20240507 08:12:21.781196 5562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f7b880ca1d444e8889207ff486edb044. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.781965 5555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe6cce5ec0a74b6987e4466e1b7d964a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.783038 5841 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35094 (request call id 1408) took 1253 ms. Trace:
I20240507 08:12:21.783078 5549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85018a1ab8b84d579c5ccec679d2f5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.783121 5841 rpcz_store.cc:276] 0507 08:12:20.529865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:20.529921 (+ 56us) service_pool.cc:225] Handling call
0507 08:12:21.782933 (+1253012us) ts_tablet_manager.cc:1120] Deleting tablet 8b47c57b0f964c7894461a594bb22539
0507 08:12:21.782942 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.783023 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1252818}
I20240507 08:12:21.755676 7820 leader_election.cc:288] T 53bc9bf1c04e4015a152cb750184c9ce P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.786809 5841 tablet_replica.cc:330] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.787935 5841 raft_consensus.cc:2229] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.788208 5841 pending_rounds.cc:62] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.788266 5841 pending_rounds.cc:68] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924840490778624 op_type: NO_OP noop_request { }
I20240507 08:12:21.788393 5841 raft_consensus.cc:2866] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.788487 5841 raft_consensus.cc:2258] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.790931 5841 ts_tablet_manager.cc:1875] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.791867 7762 raft_consensus.cc:482] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.791919 5560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0d15ec7033c4647b10e4085a097eebb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240507 08:12:21.791991 7762 raft_consensus.cc:504] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.792544 7762 leader_election.cc:288] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
W20240507 08:12:21.792977 5839 thread.cc:636] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.012s sys 0.007s
W20240507 08:12:21.793085 5839 thread.cc:602] log-alloc [worker] (thread pool) Time spent starting thread: real 0.529s user 0.012s sys 0.007s
I20240507 08:12:21.793882 5839 tablet_bootstrap.cc:492] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:21.794070 5839 ts_tablet_manager.cc:1367] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.537s user 0.015s sys 0.007s
I20240507 08:12:21.796602 5841 ts_tablet_manager.cc:1888] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.796700 5841 log.cc:1196] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/d4fac82e12d0437e950d01dbc4117efa
I20240507 08:12:21.797009 5841 ts_tablet_manager.cc:1909] T d4fac82e12d0437e950d01dbc4117efa P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.801175 6513 raft_consensus.cc:2229] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.801507 6513 raft_consensus.cc:2258] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.802044 6513 ts_tablet_manager.cc:1875] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.781810 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet b31fdadee60e4f5e94e92c536e18774d (table default.Table_f993482dcb444626a4d006bf1755dcca [id=aa4bf50ad570490b8120f3d75aad746b]) successfully deleted
W20240507 08:12:21.804018 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 8b47c57b0f964c7894461a594bb22539 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b47c57b0f964c7894461a594bb22539 already in progress: creating tablet
W20240507 08:12:21.804222 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1cbd9a6d4b484b8a9f7c722a8bdde027 already in progress: creating tablet
W20240507 08:12:21.770218 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet d3f120d9fbf542d39508903883ecb5e6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3f120d9fbf542d39508903883ecb5e6 already in progress: creating tablet
I20240507 08:12:21.808384 7820 raft_consensus.cc:482] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.808521 7820 raft_consensus.cc:504] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:21.705045 5654 consensus_peers.cc:485] T 89db2847c09b4c74b809bd13f78d0b8b P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 89db2847c09b4c74b809bd13f78d0b8b. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:21.809042 5654 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.246.194:41443, user_credentials={real_user=slave}} blocked reactor thread for 104019us
I20240507 08:12:21.809113 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3769eb670ff442e2ad2fa5b6a934512a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.809691 5579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.810083 5682 tablet_service.cc:1514] Processing DeleteTablet for tablet 43644b4405f44bfd9a1afcaedbd68db5 with delete_type TABLET_DATA_DELETED (replaced by tablet 192492075dbf4badb8c70a9546ddd80a) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:21.810132 5654 leader_election.cc:341] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.810110 5697 tablet_service.cc:1514] Processing DeleteTablet for tablet b069d601b8fb4e75934675e7996a683a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.810464 5685 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.811038 5690 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.812171 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce11e4e46b345dba1e5e1a6f1af3dd4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.812808 7720 raft_consensus.cc:482] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240507 08:12:21.812924 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 310c601abead42e0bd2a2df1a0e16ec2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 310c601abead42e0bd2a2df1a0e16ec2 already in progress: creating tablet
I20240507 08:12:21.812948 7720 raft_consensus.cc:504] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:21.813279 5378 leader_election.cc:341] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:21.813452 5378 leader_election.cc:341] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.813550 7720 leader_election.cc:288] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.813694 5378 consensus_peers.cc:485] T a605be6967a041388dfe81053fbb602e P dc665657d5b34fd7aec884e0f2668e11 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.813462 5841 tablet_replica.cc:330] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
W20240507 08:12:21.813829 5378 leader_election.cc:341] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.813979 5841 raft_consensus.cc:2229] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.814124 5841 raft_consensus.cc:2258] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.814699 5841 ts_tablet_manager.cc:1875] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.814931 5520 leader_election.cc:341] T b8ebbcf0d5a54d63a96a9374e839af83 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: b8ebbcf0d5a54d63a96a9374e839af83
W20240507 08:12:21.815351 5520 leader_election.cc:341] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 99e4514039304f0696c69a057db0ea6e
I20240507 08:12:21.815207 7820 leader_election.cc:288] T 99e4514039304f0696c69a057db0ea6e P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.818899 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ece0e5eae0d949149dd3bd3f4bbd4ca7 already in progress: creating tablet
W20240507 08:12:21.820338 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet e797fee2c43c4461a5cb9e2f0a2bb0fc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e797fee2c43c4461a5cb9e2f0a2bb0fc already in progress: creating tablet
I20240507 08:12:21.821723 7820 raft_consensus.cc:482] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.829875 7649 raft_consensus.cc:482] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.830062 7649 raft_consensus.cc:504] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.833038 5563 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6b0ccf73cc46a3a7815e5d09ed7614 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.833214 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1324) took 2282 ms. Trace:
I20240507 08:12:21.833711 7649 leader_election.cc:288] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.833768 7079 rpcz_store.cc:276] 0507 08:12:19.502022 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.502070 (+ 48us) service_pool.cc:225] Handling call
0507 08:12:21.769185 (+2267115us) ts_tablet_manager.cc:1120] Deleting tablet b82da7a22025499b928406579ae43a50
0507 08:12:21.769198 (+ 13us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.784767 (+ 15569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266919}
I20240507 08:12:21.834136 7079 tablet_replica.cc:330] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
I20240507 08:12:21.834352 7079 raft_consensus.cc:2229] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.834486 7079 raft_consensus.cc:2258] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.834964 7079 ts_tablet_manager.cc:1875] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.835341 5843 tablet_bootstrap.cc:492] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.835686 5556 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ab1e02d0eb44bdbbf2b3067f4ffdd5f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:21.835930 5843 ts_tablet_manager.cc:1367] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.319s user 0.053s sys 0.044s
I20240507 08:12:21.836568 5843 raft_consensus.cc:348] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.836841 5843 raft_consensus.cc:374] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.836939 5843 raft_consensus.cc:729] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.835739 5841 ts_tablet_manager.cc:1888] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.837199 5841 log.cc:1196] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/c1f3dc0a02c948e7ad05ddde8f2d58c1
I20240507 08:12:21.837518 5841 ts_tablet_manager.cc:1909] T c1f3dc0a02c948e7ad05ddde8f2d58c1 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
W20240507 08:12:21.838014 5377 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.246.195:44165, user_credentials={real_user=slave}} blocked reactor thread for 220666us
I20240507 08:12:21.838382 5843 consensus_queue.cc:260] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.839489 5377 leader_election.cc:302] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.839761 7762 raft_consensus.cc:2781] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.839838 7762 raft_consensus.cc:482] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.839885 7762 raft_consensus.cc:3037] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.842244 5841 tablet_replica.cc:330] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.842206 7762 raft_consensus.cc:504] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.842428 5843 ts_tablet_manager.cc:1398] T f513573e340547d6ae5ab848d00ace72 P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240507 08:12:21.842645 5843 tablet_bootstrap.cc:492] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.842778 7762 leader_election.cc:288] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.843016 5841 raft_consensus.cc:2229] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:21.843214 5841 pending_rounds.cc:62] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Trying to abort 1 pending ops.
I20240507 08:12:21.843269 5841 pending_rounds.cc:68] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924839601639424 op_type: NO_OP noop_request { }
I20240507 08:12:21.843369 5841 raft_consensus.cc:2866] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.843466 5841 raft_consensus.cc:2258] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.848059 5841 ts_tablet_manager.cc:1875] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.850008 5843 tablet_bootstrap.cc:654] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.855957 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3769eb670ff442e2ad2fa5b6a934512a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.821875 7820 raft_consensus.cc:504] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.856806 7820 leader_election.cc:288] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
W20240507 08:12:21.860751 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet b82da7a22025499b928406579ae43a50 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b82da7a22025499b928406579ae43a50 already in progress: creating tablet
I20240507 08:12:21.860837 5841 ts_tablet_manager.cc:1888] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.860919 5841 log.cc:1196] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/093fb3986a674b93a435408c7ec85500
I20240507 08:12:21.861234 5841 ts_tablet_manager.cc:1909] T 093fb3986a674b93a435408c7ec85500 P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.862870 6513 ts_tablet_manager.cc:1888] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:21.862977 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.863394 5843 tablet_bootstrap.cc:492] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.863466 6513 log.cc:1196] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/08aac6e88b7c41d8a59d1e12e7f6c14e
W20240507 08:12:21.863572 5654 leader_election.cc:341] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
W20240507 08:12:21.863884 5520 leader_election.cc:341] T 14929efb0ed049d6b554598a5b8cb494 P fe673739d8084e7da5d7edd722e857d8 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Not found: Tablet not found: 14929efb0ed049d6b554598a5b8cb494
I20240507 08:12:21.863941 6513 ts_tablet_manager.cc:1909] T 08aac6e88b7c41d8a59d1e12e7f6c14e P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.864141 5843 ts_tablet_manager.cc:1367] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.003s
I20240507 08:12:21.864254 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1247) took 5302 ms. Trace:
I20240507 08:12:21.864341 6513 rpcz_store.cc:276] 0507 08:12:16.562078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.563145 (+ 1067us) service_pool.cc:225] Handling call
0507 08:12:21.718306 (+5155161us) ts_tablet_manager.cc:1120] Deleting tablet 08aac6e88b7c41d8a59d1e12e7f6c14e
0507 08:12:21.727832 (+ 9526us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.845845 (+118013us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.862766 (+ 16921us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.864237 (+ 1471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":46409,"tablet-delete.queue_time_us":5155038}
I20240507 08:12:21.864432 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2ed8516944a083c02e07045e44bd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:21.864681 6513 tablet_replica.cc:330] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.864874 6513 raft_consensus.cc:2229] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
W20240507 08:12:21.864897 5378 leader_election.cc:341] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.865197 6513 pending_rounds.cc:62] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Trying to abort 1 pending ops.
I20240507 08:12:21.865265 6513 pending_rounds.cc:68] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7024924824412463104 op_type: NO_OP noop_request { }
I20240507 08:12:21.865392 6513 raft_consensus.cc:2866] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20240507 08:12:21.865487 6513 raft_consensus.cc:2258] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.866015 5843 raft_consensus.cc:348] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.866734 5839 raft_consensus.cc:348] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.867007 5839 raft_consensus.cc:374] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.867146 5839 raft_consensus.cc:729] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.867483 5839 consensus_queue.cc:260] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.867585 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43a80d38ed4788a13421b245fe6e8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.868026 5442 raft_consensus.cc:2454] T 1f43a80d38ed4788a13421b245fe6e8b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:21.868409 5843 raft_consensus.cc:374] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.868530 5843 raft_consensus.cc:729] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.868760 5843 consensus_queue.cc:260] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.869299 5843 ts_tablet_manager.cc:1398] T 53fb38b567c5455ebc441e5f71db694f P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240507 08:12:21.870286 5442 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005100f64fd49a3ac29d34aa728fdfb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.870517 5442 raft_consensus.cc:2454] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:21.871187 6513 ts_tablet_manager.cc:1875] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.872124 5843 tablet_bootstrap.cc:492] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.873966 7762 raft_consensus.cc:482] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.874096 7762 raft_consensus.cc:504] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.874676 7762 leader_election.cc:288] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.875504 5841 tablet_replica.cc:330] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: stopping tablet replica
I20240507 08:12:21.875900 5841 raft_consensus.cc:2229] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.876047 5841 raft_consensus.cc:2258] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.876513 5841 ts_tablet_manager.cc:1875] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.878546 5843 tablet_bootstrap.cc:654] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.871187 5840 ts_tablet_manager.cc:1398] T 53fb38b567c5455ebc441e5f71db694f P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.334s user 0.005s sys 0.016s
I20240507 08:12:21.880628 5579 raft_consensus.cc:3037] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.880759 5840 tablet_bootstrap.cc:492] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
I20240507 08:12:21.882028 5840 tablet_bootstrap.cc:654] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.884095 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a535a1039d43b9b0bf2c61304c2d01" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:21.885628 5378 leader_election.cc:341] T 55a535a1039d43b9b0bf2c61304c2d01 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.890671 6513 ts_tablet_manager.cc:1888] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.890784 6513 log.cc:1196] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/1ab94b1a09dd4969b928b017229ce4bf
I20240507 08:12:21.890941 5843 tablet_bootstrap.cc:492] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11: No bootstrap required, opened a new log
I20240507 08:12:21.891093 5843 ts_tablet_manager.cc:1367] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.001s
I20240507 08:12:21.891166 6513 ts_tablet_manager.cc:1909] T 1ab94b1a09dd4969b928b017229ce4bf P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.891459 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1250) took 5246 ms. Trace:
I20240507 08:12:21.891536 6513 rpcz_store.cc:276] 0507 08:12:16.644625 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.645897 (+ 1272us) service_pool.cc:225] Handling call
0507 08:12:21.864633 (+5218736us) ts_tablet_manager.cc:1120] Deleting tablet 1ab94b1a09dd4969b928b017229ce4bf
0507 08:12:21.864645 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.879598 (+ 14953us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.890583 (+ 10985us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.891443 (+ 860us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5218142}
I20240507 08:12:21.891820 6513 tablet_replica.cc:330] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.891765 5843 raft_consensus.cc:348] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.891940 5843 raft_consensus.cc:374] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.891995 5843 raft_consensus.cc:729] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.892166 5843 consensus_queue.cc:260] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.894246 5446 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc665657d5b34fd7aec884e0f2668e11" is_pre_election: true
I20240507 08:12:21.894421 5446 raft_consensus.cc:2454] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cbb887175f345ada2433adaec906fd4 in term 0.
I20240507 08:12:21.892009 6513 raft_consensus.cc:2229] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.897660 6513 raft_consensus.cc:2258] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.898175 6513 ts_tablet_manager.cc:1875] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.899339 5839 ts_tablet_manager.cc:1398] T 385f4c1eb7f4493a924b813734ddcfdb P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.082s user 0.002s sys 0.000s
I20240507 08:12:21.899798 5839 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
W20240507 08:12:21.899927 5654 consensus_peers.cc:485] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.906167 7854 raft_consensus.cc:482] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.906514 7854 raft_consensus.cc:504] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
W20240507 08:12:21.907094 5656 consensus_peers.cc:485] T eec3704b135740f1a906d60f3ea87ffe P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eec3704b135740f1a906d60f3ea87ffe. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:21.907752 5656 consensus_peers.cc:485] T 6def4bf6a7554a4090a39b1823c4097c P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6def4bf6a7554a4090a39b1823c4097c. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.908464 5656 leader_election.cc:302] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.908811 7649 raft_consensus.cc:2781] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.908967 7649 raft_consensus.cc:482] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.909085 7649 raft_consensus.cc:3037] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.910948 5843 ts_tablet_manager.cc:1398] T fccc127d380d4035905c1b152ab1017b P dc665657d5b34fd7aec884e0f2668e11: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240507 08:12:21.912184 7649 raft_consensus.cc:504] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.913064 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f43a80d38ed4788a13421b245fe6e8b" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.913666 5654 leader_election.cc:341] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.913756 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddb7dd28f43488581d3a75bf1d68a8a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:21.913878 5841 ts_tablet_manager.cc:1888] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.913980 5841 log.cc:1196] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-0/wal/wals/8005100f64fd49a3ac29d34aa728fdfb
W20240507 08:12:21.914273 5378 leader_election.cc:341] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.914644 7854 leader_election.cc:288] T 0ddb7dd28f43488581d3a75bf1d68a8a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.919225 5839 tablet_bootstrap.cc:654] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.922544 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.922735 5720 raft_consensus.cc:2454] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:21.924799 5720 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ce11e4e46b345dba1e5e1a6f1af3dd4" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.925525 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet b069d601b8fb4e75934675e7996a683a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
W20240507 08:12:21.927603 5654 consensus_peers.cc:485] T 5ff69b3775d94e5da1f9f327c759abb8 P 1cbb887175f345ada2433adaec906fd4 -> Peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Couldn't send request to peer fe673739d8084e7da5d7edd722e857d8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20240507 08:12:21.928066 5656 consensus_peers.cc:485] T 3c7f072f55a94cca938aa063637af9a5 P 1cbb887175f345ada2433adaec906fd4 -> Peer dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): Couldn't send request to peer dc665657d5b34fd7aec884e0f2668e11. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c7f072f55a94cca938aa063637af9a5. This is attempt 1: this message will repeat every 5th retry.
I20240507 08:12:21.929039 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2ed8516944a083c02e07045e44bd" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4"
I20240507 08:12:21.929204 5713 raft_consensus.cc:3037] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.929312 7649 leader_election.cc:288] T 1f43a80d38ed4788a13421b245fe6e8b P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.929062 5656 leader_election.cc:302] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.929917 7851 raft_consensus.cc:2781] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.929996 7851 raft_consensus.cc:482] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.930049 7851 raft_consensus.cc:3037] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.931702 5713 raft_consensus.cc:2454] T 2fdc2ed8516944a083c02e07045e44bd P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 1.
I20240507 08:12:21.932365 7851 raft_consensus.cc:504] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.933063 7851 leader_election.cc:288] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Requested vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:21.934108 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8005100f64fd49a3ac29d34aa728fdfb" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.934545 5654 leader_election.cc:341] T 8005100f64fd49a3ac29d34aa728fdfb P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.934995 5713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55a535a1039d43b9b0bf2c61304c2d01" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.935159 5713 raft_consensus.cc:2454] T 55a535a1039d43b9b0bf2c61304c2d01 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc665657d5b34fd7aec884e0f2668e11 in term 0.
I20240507 08:12:21.935173 5716 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ddb7dd28f43488581d3a75bf1d68a8a" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cbb887175f345ada2433adaec906fd4" is_pre_election: true
I20240507 08:12:21.914286 5841 ts_tablet_manager.cc:1909] T 8005100f64fd49a3ac29d34aa728fdfb P dc665657d5b34fd7aec884e0f2668e11: Deleting consensus metadata
I20240507 08:12:21.938295 6513 ts_tablet_manager.cc:1888] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:21.938977 6513 log.cc:1196] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/242ea35c61404a91b1211c5ad9207728
I20240507 08:12:21.939690 6513 ts_tablet_manager.cc:1909] T 242ea35c61404a91b1211c5ad9207728 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.941187 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1251) took 5294 ms. Trace:
I20240507 08:12:21.942656 6513 rpcz_store.cc:276] 0507 08:12:16.646968 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.647337 (+ 369us) service_pool.cc:225] Handling call
0507 08:12:21.891786 (+5244449us) ts_tablet_manager.cc:1120] Deleting tablet 242ea35c61404a91b1211c5ad9207728
0507 08:12:21.891792 (+ 6us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.913631 (+ 21839us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.938182 (+ 24551us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.941170 (+ 2988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5244157}
I20240507 08:12:21.942978 7762 raft_consensus.cc:482] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:21.943126 7762 raft_consensus.cc:504] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } }
I20240507 08:12:21.943768 7762 leader_election.cc:288] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165)
I20240507 08:12:21.943792 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1252) took 5291 ms. Trace:
I20240507 08:12:21.944615 5584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "279e95d9804c44eaa2c23a4ec02c84ff" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
W20240507 08:12:21.945608 5378 leader_election.cc:341] T 279e95d9804c44eaa2c23a4ec02c84ff P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.943864 6513 rpcz_store.cc:276] 0507 08:12:16.652748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.653664 (+ 916us) service_pool.cc:225] Handling call
0507 08:12:21.943678 (+5290014us) ts_tablet_manager.cc:1120] Deleting tablet 73f82413951c48a89e7db8fe9e50c749
0507 08:12:21.943690 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.943778 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5289548}
W20240507 08:12:21.946712 5377 leader_election.cc:341] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.947535 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1253) took 5281 ms. Trace:
I20240507 08:12:21.947250 5377 leader_election.cc:302] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.948146 7854 raft_consensus.cc:2726] T 3769eb670ff442e2ad2fa5b6a934512a P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.947618 6513 rpcz_store.cc:276] 0507 08:12:16.666358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.667858 (+ 1500us) service_pool.cc:225] Handling call
0507 08:12:21.947426 (+5279568us) ts_tablet_manager.cc:1120] Deleting tablet 9669c1169b5244218383f7aa7fbc3b85
0507 08:12:21.947445 (+ 19us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.947521 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5279409}
I20240507 08:12:21.949721 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1254) took 5283 ms. Trace:
I20240507 08:12:21.950995 6513 rpcz_store.cc:276] 0507 08:12:16.666730 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.670897 (+ 4167us) service_pool.cc:225] Handling call
0507 08:12:21.949635 (+5278738us) ts_tablet_manager.cc:1120] Deleting tablet 994f50ea19a2445288663b7d7b7ad291
0507 08:12:21.949643 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.949708 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5278585}
I20240507 08:12:21.951699 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1260) took 5192 ms. Trace:
I20240507 08:12:21.952301 6513 rpcz_store.cc:276] 0507 08:12:16.758845 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.760294 (+ 1449us) service_pool.cc:225] Handling call
0507 08:12:21.951602 (+5191308us) ts_tablet_manager.cc:1120] Deleting tablet 17c6f752a0d4485e9606a5570a0d6c96
0507 08:12:21.951612 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.951686 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5191116}
I20240507 08:12:21.952945 6513 tablet_replica.cc:330] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.953497 6513 raft_consensus.cc:2229] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.954200 6513 raft_consensus.cc:2258] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.955794 6513 ts_tablet_manager.cc:1875] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:21.956341 5377 leader_election.cc:302] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
W20240507 08:12:21.956831 5377 leader_election.cc:341] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.956991 5377 leader_election.cc:302] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dc665657d5b34fd7aec884e0f2668e11; no voters: 1cbb887175f345ada2433adaec906fd4, fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.957455 7762 raft_consensus.cc:2781] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.957482 5377 leader_election.cc:302] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.957633 7762 raft_consensus.cc:482] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.957860 7762 raft_consensus.cc:3037] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.961796 5843 tablet_bootstrap.cc:492] T aa806508ec3e489a9b2c2fa9196198d5 P dc665657d5b34fd7aec884e0f2668e11: Bootstrap starting.
I20240507 08:12:21.963631 5843 tablet_bootstrap.cc:654] T aa806508ec3e489a9b2c2fa9196198d5 P dc665657d5b34fd7aec884e0f2668e11: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.963688 7079 ts_tablet_manager.cc:1888] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.963912 7079 log.cc:1196] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-1/wal/wals/cd4379b6fded4a52843f55f27ae67121
I20240507 08:12:21.963529 7762 raft_consensus.cc:504] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.965667 7079 ts_tablet_manager.cc:1909] T cd4379b6fded4a52843f55f27ae67121 P fe673739d8084e7da5d7edd722e857d8: Deleting consensus metadata
I20240507 08:12:21.965935 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "dc665657d5b34fd7aec884e0f2668e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
W20240507 08:12:21.966470 5378 leader_election.cc:341] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.966835 5656 leader_election.cc:302] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1cbb887175f345ada2433adaec906fd4, dc665657d5b34fd7aec884e0f2668e11; no voters: fe673739d8084e7da5d7edd722e857d8
I20240507 08:12:21.967132 7851 raft_consensus.cc:2781] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240507 08:12:21.967141 7762 leader_election.cc:288] T 01a9165fdc41482cb1f8431ab6479126 P dc665657d5b34fd7aec884e0f2668e11 [CANDIDATE]: Term 1 election: Requested vote from peers 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165), fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443)
I20240507 08:12:21.967257 7851 raft_consensus.cc:482] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240507 08:12:21.967344 7851 raft_consensus.cc:3037] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Advancing to term 1
I20240507 08:12:21.967566 7762 raft_consensus.cc:2726] T 5ce11e4e46b345dba1e5e1a6f1af3dd4 P dc665657d5b34fd7aec884e0f2668e11 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240507 08:12:21.969246 5840 tablet_bootstrap.cc:492] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4: No bootstrap required, opened a new log
I20240507 08:12:21.969367 5840 ts_tablet_manager.cc:1367] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.011s
I20240507 08:12:21.969642 7854 raft_consensus.cc:2781] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 FOLLOWER]: Leader election won for term 1
I20240507 08:12:21.969981 7854 raft_consensus.cc:686] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [term 1 LEADER]: Becoming Leader. State: Replica: dc665657d5b34fd7aec884e0f2668e11, State: Running, Role: LEADER
I20240507 08:12:21.970160 5840 raft_consensus.cc:348] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.970324 5840 raft_consensus.cc:374] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:21.970373 5840 raft_consensus.cc:729] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cbb887175f345ada2433adaec906fd4, State: Initialized, Role: FOLLOWER
I20240507 08:12:21.970646 5840 consensus_queue.cc:260] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.970721 7851 raft_consensus.cc:504] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.971113 5840 ts_tablet_manager.cc:1398] T 190baa48859640cf928642dccbebfceb P 1cbb887175f345ada2433adaec906fd4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240507 08:12:21.970772 7854 consensus_queue.cc:237] T 2fdc2ed8516944a083c02e07045e44bd P dc665657d5b34fd7aec884e0f2668e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:21.971508 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01a9165fdc41482cb1f8431ab6479126" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8"
I20240507 08:12:21.971751 5840 tablet_bootstrap.cc:492] T 5c49745fce2949d1909baf7583b157a7 P 1cbb887175f345ada2433adaec906fd4: Bootstrap starting.
W20240507 08:12:21.972021 5654 leader_election.cc:341] T 01a9165fdc41482cb1f8431ab6479126 P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:21.972779 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 08aac6e88b7c41d8a59d1e12e7f6c14e (table default.Table_a314de990d004c20810e4ce3cc8c22dd [id=33eb2f3eadb1433a833d373add05496c]) successfully deleted
I20240507 08:12:21.972991 5840 tablet_bootstrap.cc:654] T 5c49745fce2949d1909baf7583b157a7 P 1cbb887175f345ada2433adaec906fd4: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:21.973017 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 1ab94b1a09dd4969b928b017229ce4bf (table default.Table_3ca80faa43124b57991879a94bd046cd [id=d26d851d0f5244419cc308132e26e5e1]) successfully deleted
I20240507 08:12:21.973181 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet 242ea35c61404a91b1211c5ad9207728 (table default.Table_5d4ba83817d04005be08381a5c9c15b3 [id=a79c79866e6547a2b0ea94eeaec8c98e]) successfully deleted
W20240507 08:12:21.973367 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 73f82413951c48a89e7db8fe9e50c749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 73f82413951c48a89e7db8fe9e50c749 already in progress: creating tablet
W20240507 08:12:21.973609 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 9669c1169b5244218383f7aa7fbc3b85 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9669c1169b5244218383f7aa7fbc3b85 already in progress: creating tablet
W20240507 08:12:21.973773 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 994f50ea19a2445288663b7d7b7ad291 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 994f50ea19a2445288663b7d7b7ad291 already in progress: creating tablet
W20240507 08:12:21.973897 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 17c6f752a0d4485e9606a5570a0d6c96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 17c6f752a0d4485e9606a5570a0d6c96 already in progress: creating tablet
I20240507 08:12:21.974591 6513 ts_tablet_manager.cc:1888] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20240507 08:12:21.974628 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1326) took 2187 ms. Trace:
I20240507 08:12:21.974857 6513 log.cc:1196] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/bee9e434083f4fa6859cfe2b7cca16e5
I20240507 08:12:21.974766 7079 rpcz_store.cc:276] 0507 08:12:19.786693 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.786742 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:21.834079 (+2047337us) ts_tablet_manager.cc:1120] Deleting tablet cd4379b6fded4a52843f55f27ae67121
0507 08:12:21.834091 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.861199 (+ 27108us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.963546 (+102347us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.974611 (+ 11065us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2047154}
W20240507 08:12:21.804390 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 35678404b67b4273b61fa5f922152fe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35678404b67b4273b61fa5f922152fe3 already in progress: creating tablet
W20240507 08:12:21.975339 5251 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.246.193:45985, user_credentials={real_user=slave}} blocked reactor thread for 170929us
I20240507 08:12:21.975376 5249 catalog_manager.cc:4830] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): tablet cd4379b6fded4a52843f55f27ae67121 (table default.Table_e61dfd2fdddc4cf18a97f9c3a617ad39 [id=1cdc664431304379a6bf38c5208cf108]) successfully deleted
I20240507 08:12:21.975482 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1327) took 2168 ms. Trace:
I20240507 08:12:21.975135 6513 ts_tablet_manager.cc:1909] T bee9e434083f4fa6859cfe2b7cca16e5 P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:21.975590 7079 rpcz_store.cc:276] 0507 08:12:19.807418 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.807477 (+ 59us) service_pool.cc:225] Handling call
0507 08:12:21.975248 (+2167771us) ts_tablet_manager.cc:1120] Deleting tablet 4bd0f775bfa344b9b02afc875db46af7
0507 08:12:21.975256 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.975468 (+ 212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2167556}
I20240507 08:12:21.976133 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1261) took 5194 ms. Trace:
I20240507 08:12:21.976203 6513 rpcz_store.cc:276] 0507 08:12:16.782071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.782218 (+ 147us) service_pool.cc:225] Handling call
0507 08:12:21.952905 (+5170687us) ts_tablet_manager.cc:1120] Deleting tablet bee9e434083f4fa6859cfe2b7cca16e5
0507 08:12:21.952914 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.966104 (+ 13190us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.974508 (+ 8404us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:21.976118 (+ 1610us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5170325}
I20240507 08:12:21.976553 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1265) took 5158 ms. Trace:
I20240507 08:12:21.977530 6513 rpcz_store.cc:276] 0507 08:12:16.818311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.818360 (+ 49us) service_pool.cc:225] Handling call
0507 08:12:21.976458 (+5158098us) ts_tablet_manager.cc:1120] Deleting tablet a68badb3b5644ed8b007b2b04bc4f6e1
0507 08:12:21.976465 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.976540 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5157903}
I20240507 08:12:21.977882 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1266) took 5091 ms. Trace:
I20240507 08:12:21.977950 6513 rpcz_store.cc:276] 0507 08:12:16.886487 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.886541 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:21.977756 (+5091215us) ts_tablet_manager.cc:1120] Deleting tablet 219bad2b972c45a8930d6f47ca361858
0507 08:12:21.977777 (+ 21us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.977849 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5091047}
I20240507 08:12:21.978219 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1269) took 5077 ms. Trace:
I20240507 08:12:21.978296 6513 rpcz_store.cc:276] 0507 08:12:16.900315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.901132 (+ 817us) service_pool.cc:225] Handling call
0507 08:12:21.978133 (+5077001us) ts_tablet_manager.cc:1120] Deleting tablet 3954a4e0d3104161aad427eb2781d55d
0507 08:12:21.978140 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.978205 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5076846}
I20240507 08:12:21.978510 6513 tablet_replica.cc:330] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:21.978672 6513 raft_consensus.cc:2229] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240507 08:12:21.978801 6513 raft_consensus.cc:2258] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.979274 6513 ts_tablet_manager.cc:1875] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.979108 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 38ce7a166001486197b78f8b02a437ed because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38ce7a166001486197b78f8b02a437ed already in progress: creating tablet
W20240507 08:12:21.983089 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 372dd1ea1a4043c9ad5afb1d48dfc133 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 372dd1ea1a4043c9ad5afb1d48dfc133 already in progress: creating tablet
I20240507 08:12:21.983436 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet d4fac82e12d0437e950d01dbc4117efa (table default.Table_e4cc0dc4d852467c9bdeaeb440199da8 [id=e3fb663ef2f44f9f838494aa1561e9a9]) successfully deleted
W20240507 08:12:21.983633 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 35678404b67b4273b61fa5f922152fe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35678404b67b4273b61fa5f922152fe3 already in progress: creating tablet
W20240507 08:12:21.983808 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet c2a3151ab3214a1981f378903bb86a2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2a3151ab3214a1981f378903bb86a2d already in progress: creating tablet
I20240507 08:12:21.984395 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet c1f3dc0a02c948e7ad05ddde8f2d58c1 (table default.Table_02a149a05852404a912a65ef0c6bc205 [id=1d33bf3079104056bfdd3b875017f483]) successfully deleted
W20240507 08:12:21.984591 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 99a243c9597348bb96f10f8481200a7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99a243c9597348bb96f10f8481200a7d already in progress: creating tablet
W20240507 08:12:21.984616 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 4bd0f775bfa344b9b02afc875db46af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4bd0f775bfa344b9b02afc875db46af7 already in progress: creating tablet
W20240507 08:12:21.984777 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet 5ae07896c98b4d50b3cfae208b46ea76 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae07896c98b4d50b3cfae208b46ea76 already in progress: creating tablet
W20240507 08:12:21.984970 5251 catalog_manager.cc:4810] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): delete failed for tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0ed9f8f5c8b4a20a4fdb1dfe41492ac already in progress: creating tablet
I20240507 08:12:21.985122 5251 catalog_manager.cc:4830] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985): tablet 093fb3986a674b93a435408c7ec85500 (table default.Table_82aad3e9a8064c75a66f0675c60c76e1 [id=c72296a9a4f0452fa4ac3809ca07312b]) successfully deleted
I20240507 08:12:21.986822 5262 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 2edcda81004c4e7880af58a6c3967515
I20240507 08:12:21.991079 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1328) took 2182 ms. Trace:
I20240507 08:12:21.991170 7079 rpcz_store.cc:276] 0507 08:12:19.808387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.808438 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:21.990956 (+2182518us) ts_tablet_manager.cc:1120] Deleting tablet 669ffac4b4f34551ba930f0a259e78a6
0507 08:12:21.990968 (+ 12us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.991061 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182343}
I20240507 08:12:21.991504 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1329) took 2176 ms. Trace:
I20240507 08:12:21.991598 7079 rpcz_store.cc:276] 0507 08:12:19.814740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.814794 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:21.991405 (+2176611us) ts_tablet_manager.cc:1120] Deleting tablet 6e2759ebfa0840f8b21d749354768c8d
0507 08:12:21.991414 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.991490 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2176138}
I20240507 08:12:21.991904 7079 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37096 (request call id 1330) took 2169 ms. Trace:
W20240507 08:12:21.991941 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 669ffac4b4f34551ba930f0a259e78a6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 669ffac4b4f34551ba930f0a259e78a6 already in progress: creating tablet
I20240507 08:12:21.991979 7079 rpcz_store.cc:276] 0507 08:12:19.822444 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:19.822528 (+ 84us) service_pool.cc:225] Handling call
0507 08:12:21.991778 (+2169250us) ts_tablet_manager.cc:1120] Deleting tablet 92e1e4cbc3e34af8bb56fc0ae41b7377
0507 08:12:21.991789 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.991889 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2169063}
I20240507 08:12:21.992208 7079 tablet_replica.cc:330] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: stopping tablet replica
W20240507 08:12:21.992184 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 6e2759ebfa0840f8b21d749354768c8d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e2759ebfa0840f8b21d749354768c8d already in progress: creating tablet
I20240507 08:12:21.992403 7079 raft_consensus.cc:2229] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240507 08:12:21.992563 5249 catalog_manager.cc:4810] TS fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): delete failed for tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 92e1e4cbc3e34af8bb56fc0ae41b7377 already in progress: creating tablet
I20240507 08:12:21.992587 7079 raft_consensus.cc:2258] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:21.993109 7079 ts_tablet_manager.cc:1875] T cbf7dd6413fb47108cce25cde80738fe P fe673739d8084e7da5d7edd722e857d8: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240507 08:12:21.993453 5259 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 122867a66b58468f86feb5ce4282555e consensus state has no leader
I20240507 08:12:21.993829 5259 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_fb2b770b5a3b444783dbaf68a50320cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:21.994316 5259 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb2b770b5a3b444783dbaf68a50320cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:21.994664 5257 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_26149001a1a44a28865ac44532375eca" } new_table_name: "default.Table_a41c9301c0154eb8946edfb68413c383" modify_external_catalogs: true
I20240507 08:12:21.995126 5260 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 1227b63fbc6b4625bef0b37404b76a6f
I20240507 08:12:21.995527 5260 catalog_manager.cc:6554] ReplaceTablet: tablet 1227b63fbc6b4625bef0b37404b76a6f deleted and replaced by tablet 869b30eaaff64f94b502ead49d50d557
I20240507 08:12:21.995803 5260 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d608eb93e54b43d2b21ba5c735d0f778" } modify_external_catalogs: true
I20240507 08:12:21.995967 5260 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_d608eb93e54b43d2b21ba5c735d0f778" } modify_external_catalogs: true
I20240507 08:12:21.999110 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edcda81004c4e7880af58a6c3967515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:55948
I20240507 08:12:21.999321 5549 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 869b30eaaff64f94b502ead49d50d557) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:22.000228 5839 tablet_bootstrap.cc:492] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8: No bootstrap required, opened a new log
I20240507 08:12:22.000362 5839 ts_tablet_manager.cc:1367] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8: Time spent bootstrapping tablet: real 0.101s user 0.001s sys 0.002s
W20240507 08:12:22.000835 5258 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 434db44a626e426aaa2d4d055899a56b consensus state has no leader
I20240507 08:12:22.001245 5258 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_5d485c4bb1214b37933ac72bb16c6ada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:22.001842 5258 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d485c4bb1214b37933ac72bb16c6ada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:22.003019 6513 ts_tablet_manager.cc:1888] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20240507 08:12:22.003129 6513 log.cc:1196] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/c534708f1abd4a33965eea92bdc0d0de
I20240507 08:12:22.003425 5255 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_a024af77b7ff417889d0cdf90f105834"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:22.003959 5255 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a024af77b7ff417889d0cdf90f105834 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:22.005184 5264 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_d3122d3450094367b6750b98a0a5ceea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:22.005654 5264 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d3122d3450094367b6750b98a0a5ceea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:22.005677 6513 ts_tablet_manager.cc:1909] T c534708f1abd4a33965eea92bdc0d0de P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:22.005965 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1271) took 5075 ms. Trace:
I20240507 08:12:22.006034 6513 rpcz_store.cc:276] 0507 08:12:16.930229 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:16.931431 (+ 1202us) service_pool.cc:225] Handling call
0507 08:12:21.978475 (+5047044us) ts_tablet_manager.cc:1120] Deleting tablet c534708f1abd4a33965eea92bdc0d0de
0507 08:12:21.978482 (+ 7us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:21.991775 (+ 13293us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:22.002936 (+ 11161us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:22.005948 (+ 3012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5046883}
W20240507 08:12:22.006362 5261 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ac3765fa3484f88ad6fd930d289fd7a consensus state has no leader
I20240507 08:12:22.006394 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1276) took 4934 ms. Trace:
I20240507 08:12:22.006467 6513 rpcz_store.cc:276] 0507 08:12:17.071605 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.071676 (+ 71us) service_pool.cc:225] Handling call
0507 08:12:22.006283 (+4934607us) ts_tablet_manager.cc:1120] Deleting tablet c1b0a2dddb15451a8e6230ffb52d5302
0507 08:12:22.006293 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.006378 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4934416}
I20240507 08:12:22.006701 6513 tablet_replica.cc:330] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: stopping tablet replica
I20240507 08:12:22.006626 5261 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0f7492244d9a44b3998beb1d5a959eeb" } new_table_name: "default.Table_c47933f4ea5d40ff8c469541cc97d9a9" modify_external_catalogs: true
I20240507 08:12:22.006893 6513 raft_consensus.cc:2229] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 1 LEADER]: Raft consensus shutting down.
I20240507 08:12:22.007207 6513 raft_consensus.cc:2258] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240507 08:12:22.007722 6513 ts_tablet_manager.cc:1875] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240507 08:12:22.009652 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet bee9e434083f4fa6859cfe2b7cca16e5 (table default.Table_615d90859dd74c9aa813e233b63dd68e [id=0652c43a334e4f04a34ff59ca43e545a]) successfully deleted
I20240507 08:12:22.010146 5693 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 869b30eaaff64f94b502ead49d50d557) from {username='slave'} at 127.0.0.1:55948
W20240507 08:12:22.009985 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet a68badb3b5644ed8b007b2b04bc4f6e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a68badb3b5644ed8b007b2b04bc4f6e1 already in progress: creating tablet
W20240507 08:12:22.010576 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 219bad2b972c45a8930d6f47ca361858 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 219bad2b972c45a8930d6f47ca361858 already in progress: creating tablet
W20240507 08:12:22.010848 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet 3954a4e0d3104161aad427eb2781d55d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3954a4e0d3104161aad427eb2781d55d already in progress: creating tablet
I20240507 08:12:22.011085 5248 catalog_manager.cc:4830] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): tablet c534708f1abd4a33965eea92bdc0d0de (table default.Table_53e17078e5714619b86165f5e3f636ff [id=c53dd18813454358a3842aecaab31471]) successfully deleted
W20240507 08:12:22.011328 5248 catalog_manager.cc:4810] TS 1cbb887175f345ada2433adaec906fd4 (127.4.246.195:44165): delete failed for tablet c1b0a2dddb15451a8e6230ffb52d5302 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1b0a2dddb15451a8e6230ffb52d5302 already in progress: creating tablet
I20240507 08:12:22.012524 5256 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 94c202eef7f3466fa3e703644b40879d: Sending DeleteTablet for 3 replicas of tablet 665b3736350a4121ac62014a81e5669d
I20240507 08:12:22.013043 5256 catalog_manager.cc:6554] ReplaceTablet: tablet 665b3736350a4121ac62014a81e5669d deleted and replaced by tablet af43525d35d64868b3ac808d5f4a50a0
I20240507 08:12:22.013324 5256 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0dba2acabcc144e5929e28d0332ed1df" } modify_external_catalogs: true
I20240507 08:12:22.013478 5256 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_0dba2acabcc144e5929e28d0332ed1df" } modify_external_catalogs: true
I20240507 08:12:22.015697 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 665b3736350a4121ac62014a81e5669d with delete_type TABLET_DATA_DELETED (replaced by tablet af43525d35d64868b3ac808d5f4a50a0) from {username='slave'} at 127.0.0.1:35094
I20240507 08:12:22.015967 7853 raft_consensus.cc:482] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240507 08:12:22.016124 7853 raft_consensus.cc:504] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:22.016844 7853 leader_election.cc:288] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443), dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985)
I20240507 08:12:22.017354 5581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f64e338784704312b986762a143dda4c" candidate_uuid: "1cbb887175f345ada2433adaec906fd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe673739d8084e7da5d7edd722e857d8" is_pre_election: true
I20240507 08:12:22.017623 5408 tablet_service.cc:1514] Processing DeleteTablet for tablet 1227b63fbc6b4625bef0b37404b76a6f with delete_type TABLET_DATA_DELETED (replaced by tablet 869b30eaaff64f94b502ead49d50d557) from {username='slave'} at 127.0.0.1:35094
W20240507 08:12:22.017910 5654 leader_election.cc:341] T f64e338784704312b986762a143dda4c P 1cbb887175f345ada2433adaec906fd4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer fe673739d8084e7da5d7edd722e857d8 (127.4.246.194:41443): Illegal state: must be running to vote when last-logged opid is not known
I20240507 08:12:22.019289 5839 raft_consensus.cc:348] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:22.019497 5839 raft_consensus.cc:374] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240507 08:12:22.019562 5839 raft_consensus.cc:729] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe673739d8084e7da5d7edd722e857d8, State: Initialized, Role: FOLLOWER
I20240507 08:12:22.020179 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 2edcda81004c4e7880af58a6c3967515 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:21 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:22.020535 5839 consensus_queue.cc:260] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } } peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } }
I20240507 08:12:22.031970 6513 ts_tablet_manager.cc:1888] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20240507 08:12:22.032958 6513 log.cc:1196] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Deleting WAL directory at /tmp/dist-test-taskrbFV97/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1715069505886067-5083-0/minicluster-data/ts-2/wal/wals/e17502bb4ac64f6ca9c77a3b00ab9f4a
I20240507 08:12:22.033355 6513 ts_tablet_manager.cc:1909] T e17502bb4ac64f6ca9c77a3b00ab9f4a P 1cbb887175f345ada2433adaec906fd4: Deleting consensus metadata
I20240507 08:12:22.033746 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1277) took 4954 ms. Trace:
I20240507 08:12:22.033843 6513 rpcz_store.cc:276] 0507 08:12:17.079269 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.079329 (+ 60us) service_pool.cc:225] Handling call
0507 08:12:22.006663 (+4927334us) ts_tablet_manager.cc:1120] Deleting tablet e17502bb4ac64f6ca9c77a3b00ab9f4a
0507 08:12:22.006671 (+ 8us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.019448 (+ 12777us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:22.031868 (+ 12420us) tablet_metadata.cc:641] Metadata flushed
0507 08:12:22.033724 (+ 1856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4927136}
I20240507 08:12:22.034181 5263 catalog_manager.cc:5484] T 506fb6f8445a402eb9fbc81c8019cc8b P dc665657d5b34fd7aec884e0f2668e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193). New cstate: current_term: 1 leader_uuid: "dc665657d5b34fd7aec884e0f2668e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cbb887175f345ada2433adaec906fd4" member_type: VOTER last_known_addr { host: "127.4.246.195" port: 44165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe673739d8084e7da5d7edd722e857d8" member_type: VOTER last_known_addr { host: "127.4.246.194" port: 41443 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc665657d5b34fd7aec884e0f2668e11" member_type: VOTER last_known_addr { host: "127.4.246.193" port: 45985 } health_report { overall_health: HEALTHY } } }
I20240507 08:12:22.034868 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet f64e338784704312b986762a143dda4c (table default.Table_ad611c4e3e3543fd98e93440daea8098 [id=c7918763c3e948c1a997275ef2fa4dce]). Expected version 1 got 0
I20240507 08:12:22.035100 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 55a535a1039d43b9b0bf2c61304c2d01 (table default.Table_9baae6b9dc5848a884fa67e9130222c3 [id=060d408d159e4c21877717b9f743d93d]). Expected version 2 got 0
I20240507 08:12:22.035279 5263 catalog_manager.cc:5576] TS dc665657d5b34fd7aec884e0f2668e11 (127.4.246.193:45985) does not have the latest schema for tablet 449d6c8270184ce2918cee8ce731f788 (table default.Table_2c4d993ae928415181f53571b6bf0815 [id=3764a23f5a7847c8b46385bdfaf51f20]). Expected version 1 got 0
I20240507 08:12:22.036873 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1278) took 4795 ms. Trace:
I20240507 08:12:22.036895 5553 tablet_service.cc:1514] Processing DeleteTablet for tablet 385f4c1eb7f4493a924b813734ddcfdb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-07 08:12:00 UTC) from {username='slave'} at 127.0.0.1:37096
I20240507 08:12:22.036844 5262 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44738:
name: "default.Table_e50e1815e3a2419c99d1f84b709cf7b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20240507 08:12:22.037467 5262 catalog_manager.cc:6843] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e50e1815e3a2419c99d1f84b709cf7b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20240507 08:12:22.038378 7880 tablet.cc:1722] T 0c4cd7b56d4e468eb754d9653117d023 P fe673739d8084e7da5d7edd722e857d8: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20240507 08:12:22.039988 6513 rpcz_store.cc:276] 0507 08:12:17.241808 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.245797 (+ 3989us) service_pool.cc:225] Handling call
0507 08:12:22.036757 (+4790960us) ts_tablet_manager.cc:1120] Deleting tablet fda4b1ceaf0b4f3385258f3703e4461b
0507 08:12:22.036768 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.036857 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4790788}
I20240507 08:12:22.041049 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1280) took 4648 ms. Trace:
I20240507 08:12:22.041142 6513 rpcz_store.cc:276] 0507 08:12:17.392790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.392844 (+ 54us) service_pool.cc:225] Handling call
0507 08:12:22.040886 (+4648042us) ts_tablet_manager.cc:1120] Deleting tablet 5481ddbcd31449c4b44afb87a97138bd
0507 08:12:22.040896 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.041030 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4636988}
I20240507 08:12:22.044073 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1279) took 4652 ms. Trace:
I20240507 08:12:22.044162 6513 rpcz_store.cc:276] 0507 08:12:17.391163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.391246 (+ 83us) service_pool.cc:225] Handling call
0507 08:12:22.042637 (+4651391us) ts_tablet_manager.cc:1120] Deleting tablet 3eb49743134b4f84aa6fdd9760c13504
0507 08:12:22.042647 (+ 10us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.043802 (+ 1155us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4638627}
I20240507 08:12:22.045220 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1281) took 4642 ms. Trace:
I20240507 08:12:22.045306 6513 rpcz_store.cc:276] 0507 08:12:17.403239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.403676 (+ 437us) service_pool.cc:225] Handling call
0507 08:12:22.045123 (+4641447us) ts_tablet_manager.cc:1120] Deleting tablet 71fcb6a8839840838bedb777ff1cbbcc
0507 08:12:22.045132 (+ 9us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.045204 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4641029}
I20240507 08:12:22.046344 5839 ts_tablet_manager.cc:1398] T 7db7b8c560f048f4b02febd371aa2a8c P fe673739d8084e7da5d7edd722e857d8: Time spent starting tablet: real 0.046s user 0.000s sys 0.002s
I20240507 08:12:22.046582 5839 tablet_bootstrap.cc:492] T 1ab94b1a09dd4969b928b017229ce4bf P fe673739d8084e7da5d7edd722e857d8: Bootstrap starting.
I20240507 08:12:22.047787 5839 tablet_bootstrap.cc:654] T 1ab94b1a09dd4969b928b017229ce4bf P fe673739d8084e7da5d7edd722e857d8: Neither blocks nor log segments found. Creating new log.
I20240507 08:12:22.047760 6513 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:55948 (request call id 1282) took 4572 ms. Trace:
W20240507 08:12:22.048004 5257 catalog_manager.cc:5744] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b967f56b73aa4c4eb838c42c831b9712 consensus state has no leader
I20240507 08:12:22.048434 5257 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d6a8373656e4d0da8b3529540c558af" } modify_external_catalogs: true
I20240507 08:12:22.048640 5257 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44738:
table { table_name: "default.Table_4d6a8373656e4d0da8b3529540c558af" } modify_external_catalogs: true
I20240507 08:12:22.047832 6513 rpcz_store.cc:276] 0507 08:12:17.475251 (+ 0us) service_pool.cc:168] Inserting onto call queue
0507 08:12:17.475302 (+ 51us) service_pool.cc:225] Handling call
0507 08:12:22.047660 (+4572358us) ts_tablet_manager.cc:1120] Deleting tablet 3ef4375925474e789a39a2c28fa6adfe
0507 08:12:22.047671 (+ 11us) ts_tablet_manager.cc:1040] Acquired tablet manager lock
0507 08:12:22.047744 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us"